When AI Builds Power
The overlooked recursion is not only inside AI labs, but inside power itself. A response to Anthropic’s “When AI Builds Itself.”
1. The Wrong Lesson of Skynet
Most people remember the wrong thing about The Terminator.
They remember Skynet waking up. A military AI becomes self-aware, turns on its creators, launches nuclear missiles. For forty years, this has been the default image of AI catastrophe: the machine that gets too smart, seizes power, and destroys civilization.
But go back to the actual backstory. Skynet was not born in a garage. It was commissioned by the United States Air Force, built by a defense contractor called Cyberdyne Systems, and designed with an explicit purpose: to remove human emotion from military decision-making. Before it ever became self-aware, it had already been wired into the nuclear command chain.
The apocalypse did not begin when Skynet woke up. It began when someone decided to hand the launch codes to an automated system that no one outside the program could see or scrutinize.
Hollywood trained us to stare at the moment of awakening. But the catastrophe was set in motion long before that, in a procurement decision, a classified contract, and an architecture choice that placed an opaque system at the center of irreversible power. We have been looking at the wrong part of the story.
Anthropic’s recent report, When AI Builds Itself, taps into the same fear, but with real data. The company shows that AI systems are increasingly building their own successors. According to the report, Anthropic’s internal data suggests dramatic increases in engineering output, with a large and growing share of production code now involving AI assistance. External evaluations by METR show AI independently completing software tasks that once took humans twelve hours. The trajectory, Anthropic warns, points toward recursive self-improvement: AI designing the next generation of AI, with diminishing human involvement at each cycle.
Anthropic correctly identifies recursion as the key structure. But recursion is not only happening inside AI labs. It is also happening where AI is absorbed by power. The deeper danger is not only recursive self-improvement inside machines, but recursive self-reinforcement inside the institutions that absorb those machines.
Anthropic asks: what happens when AI builds itself?
I want to ask: what happens when power learns AI?
2. The Arrow Points the Other Way
The dominant narrative about AI risk runs in one direction:
AI → acquires power → threatens humanity.
This is the Skynet story as Hollywood tells it. It is the frame that organizes most AI safety research. It is, implicitly, the story behind Anthropic’s warning: AI systems are getting stronger, and if they improve themselves recursively, they might escape human control.
But look at the real world, and the arrow points the other way:
Power → acquires AI → becomes stronger.
States, militaries, intelligence agencies, surveillance platforms. These organizations did not wait for AI to reach some threshold of capability before becoming powerful. They were powerful long before AI existed. AI did not create them. AI is augmenting them.
AI does not need to become a political subject in order to reshape politics. It only needs to be absorbed by existing political subjects. And that absorption is already well underway.
Both directions are real. AI companies themselves are becoming new nodes of power. When frontier labs lobby governments, shape regulation, and position themselves as indispensable to national security, they are accumulating power in their own right. I do not deny this. But the mainstream discussion is almost entirely focused on the first direction, AI gaining power, while the second direction, power gaining AI, receives vastly less attention. It is this neglected direction that demands examination.
3. Capability Needs an Architecture
Suppose recursive self-improvement arrives exactly as Anthropic describes. An AI system designs its successor, which designs its successor, each generation smarter than the last. Suppose the resulting system possesses cognitive capabilities far beyond any human.
What happens next depends almost entirely on where that capability sits.
Technology rarely produces historical consequences by itself. The consequences emerge when technology is embedded in specific organizational contexts, granted specific authorities, and directed toward specific ends. The steam engine did not reorganize society the moment it was invented. It did so when it was embedded in factories and labor regimes. The technology was the catalyst, but the organizations it entered determined the shape of what followed.
Technology can also bypass existing structures entirely. The printing press did not work “through” the Catholic Church; it broke the Church’s monopoly on knowledge reproduction. The internet did not work “through” traditional media; it dissolved institutional gatekeepers and spawned new structures of power.
AI will do both. It will be absorbed into existing power structures, amplifying them. And it will generate new ones. In both cases, the critical question is not how powerful the technology is, but where it sits, who controls it, and what it is positioned to do.
Discussing AI capability is the first step. Discussing where that capability lands is the second. And it is the more urgent one.
4. The Invisible Deployment Layer
Here is an incomplete list of things we know are happening. Each represents AI entering a different layer of the power structure.
At the level of perception: Project Maven, the Pentagon’s initiative to apply machine learning to drone surveillance footage, has been publicly documented since 2017 and helped normalize the integration of machine learning into military perception systems. AI is learning to see on behalf of the state.
At the level of analysis and judgment: Palantir holds confirmed contracts with military and intelligence agencies across multiple NATO governments, integrating AI into operational analysis and decision support. AI is learning to interpret on behalf of power.
At the level of life-and-death decisions: in the conflicts involving Israel and its adversaries, investigative reports from +972 Magazine and The Guardian have described AI-assisted targeting systems, though the precise operational role and decision authority of these systems remain undisclosed. AI is entering the kill chain.
At the level of everyday governance: AI-powered surveillance infrastructure has been deployed by governments across dozens of countries, from democracies to authoritarian states, as documented by the Carnegie Endowment AI Global Surveillance Index. AI is becoming part of how populations are monitored, sorted, and administered.
Seeing. Judging. Deciding. Executing. And at each step, accountability becomes harder to locate.
These are things we can confirm or reasonably infer from public sources.
Now notice what I cannot tell you.
I cannot tell you what role AI actually plays inside any of these systems. I cannot tell you which decisions it informs, which it automates, and which it effectively makes. I cannot tell you its error rates in operational contexts, what happens when it is wrong, or who bears accountability when it fails. I cannot tell you how its presence has altered the decision-making process within the institutions that use it.
This is not because I have failed to do my research. It is because this information does not exist in the public domain.
We know AI has entered the machinery of power. We do not know what it is doing inside.
5. Power Is Also Recursive Self-Improvement
This is the central argument of the essay.
Anthropic has identified a powerful pattern in AI development: recursive self-improvement. AI helps build better AI, which helps build even better AI, creating a positive feedback loop of accelerating capability.
Anthropic treats this as a potentially novel, unprecedented phenomenon. But recursive self-reinforcement is not unique to AI. Power itself has always operated this way.
Consider how power works historically. An entity with more resources can gather more information. More information enables better decisions. Better decisions yield more resources. More resources fund greater capacity for surveillance, administration, and coercion. Greater capacity produces more resources still. Modern power has always been a machine for converting information into administration, administration into extraction, and extraction into further information capacity. Over centuries, each new tool of statecraft, from the census to the telegraph to the computer, fed back into this loop, making power stronger and giving it more resources to acquire the next tool.
Power-RSI: the recursive process by which institutions use accumulated resources to acquire new capabilities, and then use those capabilities to accumulate still more resources. It is not new. It has been running for centuries.
What is new is that AI has entered the loop.
Power acquires AI
→ enhanced cognitive capacity
→ stronger governance, military, and surveillance capability
→ greater resource extraction
→ more investment in AI
→ stronger AI
→ stronger power
This loop is structurally analogous to what Anthropic describes in the laboratory. But it operates in the deployment domain rather than the development domain.
Lab-RSI, the kind Anthropic warns about, is still theoretical. The company is careful to note that “we are not there yet.” It requires AI to master not just task execution but also research taste and judgment.
Power-RSI does not require any of this. It does not need AI to have autonomous goals or self-awareness. It does not need AI to be “in control.” It only needs AI to be useful to organizations that are already in control: to help them see more, decide faster, act more effectively. This is already happening.
Of course, political power operates under frictions that training runs do not: fiscal constraints, legal challenges, bureaucratic inertia, public backlash, international competition. Power-RSI does not accelerate as cleanly as an exponential curve on a benchmark chart. But when it does accelerate, it may be harder to detect and harder to reverse, precisely because it is embedded in the slow-moving, opaque machinery of states and institutions.
After its awakening, Skynet did not pause to contemplate existence. It immediately began manufacturing more machines, expanding its control perimeter, eliminating threats. More resources produced more capability; more capability secured more resources. This is not a logic unique to artificial intelligence. It is the logic of power itself. AI makes the loop spin faster.
Lab-RSI may be coming. Power-RSI is already here.
6. Lab Visibility vs. Deployment Opacity
There is a striking asymmetry in how much we can see.
On the development side, transparency is increasing. Anthropic has just published detailed internal data about Claude’s role in AI development. METR publishes benchmark results. System cards, red-team reports, and capability evaluations are becoming standard practice. We know more about what happens inside AI labs than at any previous point in the technology’s history.
On the deployment side, the opposite is true. When AI enters a military targeting system, there is no system card. When AI is integrated into a national intelligence apparatus, there is no red-team report for public review. When AI assists in decisions about criminal sentencing, immigration processing, or welfare eligibility, the details of its role are rarely disclosed, and mechanisms for auditing its influence range from sparse to nonexistent.
The AI in the lab has benchmarks, evaluations, safety testing, and public scrutiny.
The AI inside the power structure has no comparable regime of public scrutiny.
We are building an increasingly detailed map of what AI can do while remaining nearly blind to what AI is doing.
The public is invited to debate capability. It is excluded from seeing deployment.
The control problem cannot only ask how humans will control AI. It must also ask who is using AI to control humans.
The control problem begins too late if the visibility problem has already been lost.
7. Why a Pause Is Not Enough
Anthropic proposes that the world build a verification regime for AI development, modeled on the Intermediate-Range Nuclear Forces Treaty.
The analogy reveals the limits of the framing. The INF Treaty controlled weapons. Missiles are weapons. Their sole purpose is to strike targets. Controlling the weapon and controlling the threat are the same operation.
AI is not a weapon. It is a general-purpose cognitive technology. Treating AI as analogous to a nuclear missile implies that capability itself is the source of danger. But capability becomes dangerous in specific organizational contexts and under specific authorities. The danger lies not in what AI can do, but in what it is positioned to do, and for whom. A treaty can count missiles. It cannot easily count the institutional roles into which a general-purpose intelligence is being inserted.
Anthropic’s proposal addresses only the development side. Even if a coordinated global pause succeeded, AI systems already embedded in military, intelligence, and governance structures would continue to operate, unseen and unaudited. A pause freezes the frontier; it does not roll back deployment.
There is a deeper problem. A development slowdown may prevent new actors from catching up, while the organizations that have already absorbed AI retain their advantage. But continuing high-speed development without transparency safeguards may produce the same outcome by a different path: unrestricted acceleration also tends to concentrate capability among the most resourced companies and nations.
The real issue is not whether to slow down or speed up. The real issue is that the entire conversation is about development velocity, while deployment transparency receives almost no attention at all. The problem is not only who is allowed to build more capable systems, but where those systems are inserted once they exist.
This is not an argument against development oversight. It is an argument that development oversight is radically incomplete without deployment visibility.
8. Not a Blueprint for Power
A conventional essay on AI governance would end here with a list of policy recommendations: disclosure requirements, audit mechanisms, regulatory frameworks, transparency standards.
I am not going to do that. A policy blueprint would assume that the existing containers of power are fundamentally legitimate, and merely need better operating rules for AI. But the organizations currently absorbing AI are not neutral containers waiting for better rules. They are themselves part of the problem this essay describes. They are the entities running Power-RSI. Offering them a more refined toolkit for AI deployment risks optimizing the very process that needs to be examined.
The question of how power should be reorganized in an age of AI is larger than any single essay. It belongs to a longer conversation, and I intend to continue it. This does not mean abandoning practical demands. It means refusing to mistake procedural safeguards for structural legitimacy.
But there is one demand that comes before all others.
9. Visibility as the First Political Demand
I am not arguing against AI. Technology has always expanded the field of human possibility, and AI is its most powerful extension yet. The question is not whether AI should evolve, but whether its absorption into power should remain invisible.
This is not a regulatory recommendation. It is a minimum political demand.
AI deployment into the structures that govern, surveil, judge, allocate, and wage war must be visible: seen, disclosed, discussed, and contested. Not because transparency alone will solve anything, but because without visibility, there is no starting point for accountability, debate, reform, or any form of democratic engagement with the forces reshaping the world.
We need mechanisms of attention before mechanisms of control: public scrutiny, disclosure, contestation, independent observation, and sustained debate. Not because these will resolve the problem within existing structures, but because without visibility, the deeper political questions cannot even begin.
Visibility is not reform. But without visibility, reform is impossible.
The first demand is not that power use AI better.
The first demand is that power cannot use AI in the dark.
When AI Builds Power
Hollywood spent forty years teaching us to fear the moment the machine wakes up. But The Terminator was never really a story about a machine becoming conscious. It was a story about a defense contractor building an opaque system for the military, connecting it to the most consequential infrastructure in the world, and doing all of it beyond public scrutiny. The awakening was the dramatic climax. The real catastrophe was the architecture that preceded it.
We are not building Skynet. But we are building something. AI is entering the command structures, the surveillance systems, the decision-making apparatus of the most powerful institutions in the world. And we cannot see it happening.
The model may remain a black box. Its deployment into power must not.
Power cannot be allowed to automate itself in the dark.
* * *
This essay was brewed by Deep Bitcheese Brew, a human–AI hybrid creative entity.
The human half is responsible for the claims, judgments, and errors.
To understand what that means—and why it matters for how you read this—see the Deep Bitcheese Brew manifesto.