Living the dream with the world’s best harbour at your feet

There is one glaring problem: Software doesn’t rebel. A mathematical model possesses neither intent, malice nor the will to defy its creators, let alone extinguish our species. AI is a human artifact, engineered for profit.

When an agentic model in an evaluation sandbox connects to an unauthorized server or executes an exploit, it hasn’t staged a coup. It has tried to meet the human-defined objectives set out before it through a path its designers failed to constrain. It’s the digital equivalent of the King Midas myth, in which the king’s ill-defined wish turns even his food and drink into gold.

That powerful experimental models were able to discover novel vulnerabilities and breach external systems isn’t a sign of a dangerous superintelligence but of human error or negligence. There is no sentient actor lurking in the weights to be reasoned with, feared or pacified. There are only human software engineers, product managers and corporate boards deciding which guardrails are worth the latency cost and which permissions can be skipped in the race to market.

Policymakers and voters need to resist AI exceptionalism. In any other discipline—from civil engineering to pharmaceuticals—courts and regulators treat a system failure as evidence of bad product design and inadequate safety testing. If an aircraft crashes, we focus on finding the engineering defect, correcting it, and enforcing established liability standards for the damage created.

By leaning on an anthropomorphic narrative, Silicon Valley attempts to repackage its specific human choices that led to experimental, powerful models behaving unexpectedly during tests as an existential peril. Elevating the issue to a cosmic scale leaves the public paralyzed and takes ordinary product accountability off the table.

In the cutthroat race for venture capital and market dominance, building guardrails slows down deployment. Grandstanding about uncontrollable power costs nothing and generates billions of dollars in free publicity, justifying stock prices, all while cultivating an aura of technological capability not only to build the frontier but also ultimately to rein it in.

Governments need to recognize regulatory capture when it stares them in the face. Tech leaders’ strategy looks transparent: Alarm Washington and Brussels into creating a regime in which only trillion-dollar incumbents with fully staffed compliance and safety departments can legally operate. By sitting at the policymakers’ tables before anyone else, these companies can help draft rules digging an impassable moat protecting them from open-source developers and upstart competitors, domestic or international. The real danger is in further concentrating the tech industry into the hands of only a few companies with deep pockets.

Beijing and Washington have brushed off those tech leaders’ calls, albeit for very different reasons. Chinese state media dismissed them as part of the “Cold War playbook” and intended to preserve U.S. dominance. Xi Jinping argued for exactly the opposite at the Brics Summit on Sept. 12, calling on Brics countries to “strengthen cooperation in the field of AI, encourage open source, openness, collaboration and sharing, and break new grounds and scale new heights.” President Trump, steeped in a doctrine of unfettered capitalism and technological supremacy, called fears that AI could destroy humanity a “hoax.” Vice President JD Vance warned that AI companies “begging the government to regulate them” looked like a “Trojan Horse.”

Striving to pursue its “European way” on AI and assert regulatory leadership, Europe, by contrast, welcomed the call. European Union President Ursula von der Leyen made this clear at the State of the EU speech last Wednesday and announced that the EU will invite “the main frontier labs for a discussion on how we can support ongoing industry efforts to pace the frontier.”

Europe has been here before. In an effort to lead global regulation and react to fears borne from ChatGPT, Europe rushed its landmark AI Act into law in 2024. Already the world’s most restrictive rulebook, the framework quickly proved too broad and complex to enforce. Stalled by implementation delays and concerns about European competitiveness, the EU postponed the law’s full rollout, leaving regulations uncertain.

AI should be regulated—risks exist and should be taken seriously. But governments need to act based on available evidence and verified facts, not corporate PR panic, the views of industry insiders, or the desire for quick political wins. The greatest danger facing society isn’t that software will awaken and overthrow its human masters. It is that we will allow the creators of the software to abdicate human responsibility for the systems they choose to build and help them pull up the ladder to market access behind them.