(2025-02-05) Zvim The Risk Of Gradual Disempowerment From Ai

Zvi Mowshowitz: The Risk of Gradual Disempowerment from AI. The baseline scenario as AI becomes AGI becomes ASI (artificial superintelligence), if nothing more dramatic goes wrong first and even we successfully ‘solve alignment’ of AI to a given user and developer, is the ‘gradual’ disempowerment of humanity by AIs, as we voluntarily grant them more and more power in a vicious cycle, after which AIs control the future and an ever-increasing share of its real resources. It is unlikely that humans survive it for long.

(This gradual disempowerment is far from the only way things could go horribly wrong. There are various other ways things could go horribly wrong earlier, faster and more dramatically, especially if we indeed fail at alignment of ASI on the first try.*

Gradual disempowerment it still is a major part of the problem, including in worlds that would otherwise have survived those other threats. And I don’t know of any good proposed solutions to this. All known options seem horrible, perhaps unthinkably so. This is especially true is the kind of anarchist who one rejects on principle any collective method by which humans might steer the future.
I’ve been trying to say a version of this for a while now, with little success.

Table of Contents

• We Finally Have a Good Paper.
• The Phase 2 Problem.
• Coordination is Hard.
• Even Successful Technical Solutions Do Not Solve This.
• The Six Core Claims.
• Proposed Mitigations Are Insufficient.
• The Social Contract Will Change.
• Point of No Return.
• A Shorter Summary.
• Tyler Cowen Seems To Misunderstand Two Key Points.
• Do You Feel in Charge?.
• We Will Not By Default Meaningfully ‘Own’ the AIs For Long.
• Collusion Has Nothing to Do With This.
• If Humans Do Not Successfully Collude They Lose All Control.
• The Odds Are Against Us and the Situation is Grim.

We Finally Have a Good Paper

So I’m very happy that Jan Kulveit, Raymond Douglas*, Nora Ammann, Deger Turan, David Krueger and David Duvenaud have taken a formal crack at it, and their attempt seems excellent all around*

The Phase 2 Problem

*One term I tried out for this is the ‘Phase 2’ problem.

As in, in ‘Phase 1’ we have to solve alignment, defend against sufficiently catastrophic misuse and prevent all sorts of related failure modes. If we fail at Phase 1, we lose.

If we win at Phase 1, however, we don’t win yet. We proceed to and get to play Phase 2.*

Coordination is Hard

Still, wouldn’t humans notice what’s happening and coordinate to stop it? Not necessarily. What makes this transition particularly hard to resist is that pressures on each societal system bleed into the others.

The AIs will have all the leverage, including over others that have the rest of the leverage, and also be superhumanly good at persuasion, and everything else relevant to this discussion.

Even Successful Technical Solutions Do Not Solve This