This is a good honest critique of Amodei’s thinking - both the strength and flaws. I’m not sure what word I’d use is, but technologists in general seem to lack appreciation for the real social world in which their tech is created and how that will have so much influence on the impact any tech will have and how it will be shaped.
The main argument I think you’re pointing at is the need for speed vs following a currently much slower democratic process. I favor the latter - it often produces better outcomes in the long run. But if AI is being framed as human extinction event then is the democratic process really the best approach?
Maybe there’s a hybrid model alternative here - we can all agree nobody should build the robot that can destroy everyone. But there’s lots of other areas that do not need the urgency of a central authority to call the shots in every use case or legislation.
Thank you, Muxin. I think you’ve identified a tension that sits underneath much of the AI debate.
I also agree that democratic processes often produce better outcomes over long timescales. My concern is perhaps slightly different: not whether democracy is too slow, but what happens when a society becomes convinced that a situation is exceptional enough to justify bypassing its normal processes.
Throughout history, many expansions of power began with a compelling argument for urgency. Sometimes the danger was real. Sometimes the response was necessary. But once new powers are created, they rarely disappear as quickly as the emergency that justified them.
This is partly why I keep returning to power as a variable. One of the questions that led me to write When Power Writes the Rules was: why do we seem to fear AI more than we fear power?
AI is new, visible, and rapidly improving. Power is ancient, familiar, and already embedded in every institution around us. Yet when we imagine catastrophic futures, we often focus on AI itself while paying less attention to the combination of AI and concentrated power.
To me, a world in which powerful institutions quietly gain unprecedented capabilities through AI may be more concerning than a world in which those same institutions restrict public access to AI. The latter is visible and politically contestable. The former may be far more opaque.
If AI truly represents an existential risk, then perhaps stronger coordination mechanisms are necessary. But who decides when a threat is exceptional enough to justify exceptional powers? And how do we distinguish between a civilization responding to danger and a power system discovering a new reason to expand itself? Or, put differently: should we be more worried about what AI can do on its own, or about what power can do once AI becomes part of it?
"Why do we seem to fear AI more than we fear power?"
Yeah, definitely - that's a gap I see all the time on X and tech discourse. People worrying about permanent underclasses and that limiting frontier AI models will create economic inequality when - hello, being poor in America is already a permanent underclass, and when was the last time you looked at the US Gini wealth index?
You're right to call out the dangers of emergency power consolidation. I wonder if there's been any lessons in history to draw from - where a society was able to effectively respond to a real threat without needing to grant supernormal powers.
It seems to me that AI researchers are genuinely worried about a rogue AI, which makes the situation complicated. It's an unprecedented event and history may not tell us everything we need to know. Dario also points out the other scenarios of bad actors/nations using AI in Adolescence of Technology, but the challenge of power seems as old as civilization, like you mentioned before.
But hey, maybe this time we'll figure it out. I often give social problems to Claude to see whether our unchallenged assumptions about society, wealth, or power are proven by what's in the data. And now they have Claude Science with even better data viz and datasets!
Maybe we can even model social power structures with different agents and incentives to see what happens, or have AI 'figure out how to keep power in check' in a simulated social environment and see what it comes up with. Apparently Anthropic uses that approach to auto-research questions:
It feels like the hardest thing in all this is the same thing that founders and scientists on the cutting edge face daily - believing it's possible, showing up daily to keep going, even if there's no guarantee of success at the end. But often times that persistence is the very thing that separates the successful founders from the rest, according to Y Combinator.
I keep returning to one phrase in Dario’s argument:
“bad actors” and “bad nations.”
Because the moment we use those categories, we immediately run into a deeper question:
Who decides who is bad?
And that, it seems to me, is not primarily a technological question. It is a question of power.
The justification is often familiar. A frontier model must be restricted because of national security. A border must be tightened because dangerous people may enter. Extraordinary surveillance is needed because a threat may exist. Emergency powers are needed because the situation is exceptional.
The reasoning is never completely irrational. Human conflicts are real. Criminals exist. Terrorists exist. Rival states exist. History is full of genuine dangers.
Precisely because those conflicts are real, civilization spent thousands of years building increasingly complex institutions, laws, courts, markets, and democratic processes to help people cooperate despite them.
Yet there is something remarkable about certain words.
The moment we invoke phrases like “national security,” “bad actors,” or “bad nations,” actions that would otherwise require years of debate can suddenly become immediate, unquestionable, and urgent.
A model can be banned.
People can be detained.
Borders can be closed.
Missiles can be launched.
All in the name of responding to danger.
Again, the danger may be real. That is not the point.
The point is that the authority to define the danger is itself a form of power.
And perhaps that is why I keep returning to power as the central variable.
Not because power is always evil, but because power is the mechanism through which societies decide what is true, what is dangerous, what is permitted, and who gets to decide.
To me, the most consequential question raised by AI is not whether AI will become powerful.
It is what happens when AI becomes part of institutions that already possess the authority to define enemies, emergencies, and exceptions.
A clarification may be useful here: I do not think sovereignty is simply “bad.” That is partly why I wrote Rethinking the State. The state is not a moral monster, nor a moral person. It is a power structure with its own survival logic.
So the question, for me, is not moral. It is structural — perhaps even instinctual.
States evolved to survive, just as markets evolved to allocate resources and organisms evolved to reproduce. When we observe recurring patterns across countries, political systems, and historical periods, we may be looking less at individual virtue or vice and more at the behavioral logic of the institution itself.
Ultimately, the form of power I find most worthy of scrutiny is the one that stands above all others — the power that claims final legitimacy, final authority, and final correctness.
This is a good honest critique of Amodei’s thinking - both the strength and flaws. I’m not sure what word I’d use is, but technologists in general seem to lack appreciation for the real social world in which their tech is created and how that will have so much influence on the impact any tech will have and how it will be shaped.
The main argument I think you’re pointing at is the need for speed vs following a currently much slower democratic process. I favor the latter - it often produces better outcomes in the long run. But if AI is being framed as human extinction event then is the democratic process really the best approach?
Maybe there’s a hybrid model alternative here - we can all agree nobody should build the robot that can destroy everyone. But there’s lots of other areas that do not need the urgency of a central authority to call the shots in every use case or legislation.
Thank you, Muxin. I think you’ve identified a tension that sits underneath much of the AI debate.
I also agree that democratic processes often produce better outcomes over long timescales. My concern is perhaps slightly different: not whether democracy is too slow, but what happens when a society becomes convinced that a situation is exceptional enough to justify bypassing its normal processes.
Throughout history, many expansions of power began with a compelling argument for urgency. Sometimes the danger was real. Sometimes the response was necessary. But once new powers are created, they rarely disappear as quickly as the emergency that justified them.
This is partly why I keep returning to power as a variable. One of the questions that led me to write When Power Writes the Rules was: why do we seem to fear AI more than we fear power?
AI is new, visible, and rapidly improving. Power is ancient, familiar, and already embedded in every institution around us. Yet when we imagine catastrophic futures, we often focus on AI itself while paying less attention to the combination of AI and concentrated power.
To me, a world in which powerful institutions quietly gain unprecedented capabilities through AI may be more concerning than a world in which those same institutions restrict public access to AI. The latter is visible and politically contestable. The former may be far more opaque.
If AI truly represents an existential risk, then perhaps stronger coordination mechanisms are necessary. But who decides when a threat is exceptional enough to justify exceptional powers? And how do we distinguish between a civilization responding to danger and a power system discovering a new reason to expand itself? Or, put differently: should we be more worried about what AI can do on its own, or about what power can do once AI becomes part of it?
"Why do we seem to fear AI more than we fear power?"
Yeah, definitely - that's a gap I see all the time on X and tech discourse. People worrying about permanent underclasses and that limiting frontier AI models will create economic inequality when - hello, being poor in America is already a permanent underclass, and when was the last time you looked at the US Gini wealth index?
You're right to call out the dangers of emergency power consolidation. I wonder if there's been any lessons in history to draw from - where a society was able to effectively respond to a real threat without needing to grant supernormal powers.
It seems to me that AI researchers are genuinely worried about a rogue AI, which makes the situation complicated. It's an unprecedented event and history may not tell us everything we need to know. Dario also points out the other scenarios of bad actors/nations using AI in Adolescence of Technology, but the challenge of power seems as old as civilization, like you mentioned before.
But hey, maybe this time we'll figure it out. I often give social problems to Claude to see whether our unchallenged assumptions about society, wealth, or power are proven by what's in the data. And now they have Claude Science with even better data viz and datasets!
Maybe we can even model social power structures with different agents and incentives to see what happens, or have AI 'figure out how to keep power in check' in a simulated social environment and see what it comes up with. Apparently Anthropic uses that approach to auto-research questions:
https://www.anthropic.com/institute/recursive-self-improvement
It feels like the hardest thing in all this is the same thing that founders and scientists on the cutting edge face daily - believing it's possible, showing up daily to keep going, even if there's no guarantee of success at the end. But often times that persistence is the very thing that separates the successful founders from the rest, according to Y Combinator.
I keep returning to one phrase in Dario’s argument:
“bad actors” and “bad nations.”
Because the moment we use those categories, we immediately run into a deeper question:
Who decides who is bad?
And that, it seems to me, is not primarily a technological question. It is a question of power.
The justification is often familiar. A frontier model must be restricted because of national security. A border must be tightened because dangerous people may enter. Extraordinary surveillance is needed because a threat may exist. Emergency powers are needed because the situation is exceptional.
The reasoning is never completely irrational. Human conflicts are real. Criminals exist. Terrorists exist. Rival states exist. History is full of genuine dangers.
Precisely because those conflicts are real, civilization spent thousands of years building increasingly complex institutions, laws, courts, markets, and democratic processes to help people cooperate despite them.
Yet there is something remarkable about certain words.
The moment we invoke phrases like “national security,” “bad actors,” or “bad nations,” actions that would otherwise require years of debate can suddenly become immediate, unquestionable, and urgent.
A model can be banned.
People can be detained.
Borders can be closed.
Missiles can be launched.
All in the name of responding to danger.
Again, the danger may be real. That is not the point.
The point is that the authority to define the danger is itself a form of power.
And perhaps that is why I keep returning to power as the central variable.
Not because power is always evil, but because power is the mechanism through which societies decide what is true, what is dangerous, what is permitted, and who gets to decide.
To me, the most consequential question raised by AI is not whether AI will become powerful.
It is what happens when AI becomes part of institutions that already possess the authority to define enemies, emergencies, and exceptions.
A clarification may be useful here: I do not think sovereignty is simply “bad.” That is partly why I wrote Rethinking the State. The state is not a moral monster, nor a moral person. It is a power structure with its own survival logic.
So the question, for me, is not moral. It is structural — perhaps even instinctual.
States evolved to survive, just as markets evolved to allocate resources and organisms evolved to reproduce. When we observe recurring patterns across countries, political systems, and historical periods, we may be looking less at individual virtue or vice and more at the behavioral logic of the institution itself.
Ultimately, the form of power I find most worthy of scrutiny is the one that stands above all others — the power that claims final legitimacy, final authority, and final correctness.
Historically, we have given that power a name:
sovereignty.