I invite everyone to familiarize themselves with the arguments of both camps and to separate the wheat from the chaff. Absolute certainty is impossible, but faced with the "theories of hope," it is hard not to harbor at least a reasonable doubt.
The burden of proof should lie with the frontier AI companies and permissive governments. It is up to them to prove that the chance of a major incident is lower than 0.0001%—the maximum risk threshold for nuclear reactors. (In reality, the threshold should be much stricter for AI, given that humanity's survival is at stake). If we accept the median expert estimate of a 5% risk, are we willing to play Russian roulette with the entire biosphere? Would you board a plane if you knew there was a one-in-twenty chance it would crash?
You are already on that plane. It is about to take off.
Should we hit pause? Stop development entirely?13 Form an elite "superalignment" team—a Manhattan Project for AI safety? Adopt an international treaty to set guardrails for model training, similar to uranium enrichment regulations?
In their recent book If Anyone Builds It, Everyone Dies14, Eliezer Yudkowsky and Nate Soares argue for a binding treaty between the major AI players (China, the US, and the UK—home to Google DeepMind—and perhaps France, where Mistral AI is based15). This document would be enforced by a surveillance agency capable of imposing emergency shutdowns across borders. Modeled on the nuclear non-proliferation regime, it would establish compute caps for data centers. Exceed them, and sanctions—including military intervention as a last resort—would be used to neutralize rogue infrastructure.
Yudkowsky and Soares also propose, as a desperate measure, an urgent project to biologically enhance human intelligence. Since humans are aligned by default, we would create a human superintelligence, which could presumably build an artificial superintelligence safely. But given the rapid progress of AI, would we have time to augment humans before the advent of ASI? And that is to say nothing of the ethical minefield.
Yoshua Bengio proposes a solution that remains within the realm of artificial intelligence. With "Law Zero," he aims to create a "Scientist AI" capable of predicting and preventing unaligned actions by agentic AIs. This system would possess vast research capabilities but would be devoid of agency—the primary risk factor. Bengio's bet is that CEOs and policymakers are not suicidal. If offered a safe solution that acts as a security shield around their systems, they will take it.
We, the general public, may feel we have no leverage over decisions made in the backrooms of laboratories and governments. That is true. But if we start the conversation right now in our own countries, we can lead the way. Existential risk weighs on every territory on Earth. Economic risk does too, but it is even more acute in countries outside the US and China, where a quasi-feudal dependency threatens to take hold. We must force the conversation at the national, regional, and international levels. Given the scale of the risk, it is absurd that there are no major global forums on AI safety. It is absurd that governmental and civil groups are not on a war footing to create checks and balances against the raw forces about to be unleashed.
In the United States, some elected officials are beginning to speak up, including Congressman Ro Khanna and the tireless Bernie Sanders. At 84, Sanders seems to be the politician who best grasps the stakes. He spoke with Geoffrey Hinton during a town hall and has recently given several speeches alerting the public to the economic and existential risks. Over the coming months and years, we will need strong representatives to champion the voices of those demanding safe, public-serving AI.
If AIs remain tools, confined to narrow applications, they can bring immense benefits. But if we sprint unchecked toward generality and superintelligence, we risk a bleak future—or no future at all. We risk a fate decided by a handful of billionaires driven by greed and the promise of eternal life. The Peter Thiels of the world hope to upload themselves to the cloud in a technological parody of the Apocalypse: a "Rapture" where a chosen few ascend to a silicon paradise, leaving behind a fallen humanity and an obsolete world.
Since the invention of the atomic bomb, we have known that we can destroy ourselves; we also know that we can choose otherwise through collaboration. Not every scientific discovery is desirable: who would want "mirror bacteria"—that theoretical possibility of organisms that bypass immune defenses and devour all life? Unaligned superintelligence—default superintelligence—belongs in the same category. It is highly likely to behave like a virus: duplicating and propagating at breakneck speed to maximize goals that seem absurd to us but are vital to the AI—transforming every atom in the universe into micro-helices, computronium16, or lines of code.
We will then have squandered our cosmic endowment—that theoretically accessible space-time we should have filled with fun, as Eliezer Yudkowsky put it, or with joy, as Yoshua Bengio calls for today17.
We occupy a small, dark corner of the universe. If we snuff out the joy, no one is coming to save us.
13 The movements PauseAI (pauseai.info) and StopAI (stopai.info) advocate for this.
14 Eliezer Yudkowsky and Nate Soares, If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, New York, Little, Brown and Company, 2025.
15 Though it must be said that Mistral AI is not a frontrunner in the race to ASI.
16 A term proposed in the 90s to designate a hypothetical elementary substrate for information processing.
17 The slogan of his new AI safety organization, Law Zero, is: "Preserving joy and human agency." Website: lawzero.org.