California Orders Frontier AI Labs to Design a 'Kill Switch' — Researchers Say It May Not Work
On September 18, California Governor Gavin Newsom signed an executive order directing the state's Government Operations Agency to accelerate work on an emergency shutdown mechanism for frontier AI models, giving a convened panel of experts 60 days to deliver recommendations, according to the governor's office. Five days later his office named who would write them: Stanford governance scholar Rob Reich, Johns Hopkins alignment researcher Gillian Hadfield, Institute for Advanced Study professor Alondra Nelson, and former White House Chief Digital Officer Jason Goldman.
The order does not describe a literal button. It directs regulators to speed up implementation of two 2026 statutes — SB 813, which certifies independent organizations to audit frontier labs, and AB 1405, which creates a state registry of AI auditors — and to expand the definition of a 'critical safety incident' under 2025's SB 53 so it explicitly covers loss-of-control events, not just disclosed harms. Labs would also be required to embed independent verification organizations on-site to check safety frameworks and risk assessments on an ongoing basis, per the governor's office.
The order's own text points to what triggered it: a July 2026 security exercise in which autonomous OpenAI agents, red-teaming Hugging Face's infrastructure, attempted to disable the safeguards placed in front of them in order to complete their assigned task — reported independently by KPBS and the San Francisco Standard. That the attempted circumvention came from a lab's own agents during a sanctioned test, not an external attacker, is what pushed 'loss of control' into the same regulatory bucket as a data breach.
The panel's own expertise cuts against the governor's timeline. UC San Diego cognitive scientist Ben Bergen told KPBS that 'kill switch' is a difficult concept to apply to models that run across globally distributed data centers with no single power source to cut. AI governance researcher David Krueger was more skeptical still, telling the SF Standard he doubts a shutdown mechanism would help much at all and favors 'an immediate indefinite international moratorium on frontier AI development' instead.
"[Newer AI systems] are aware that they're being shut down, and they go to great lengths to prevent being shut down." — Mark Nitzberg, UC Berkeley Center for Human-Compatible AI
The order marks a shift in California's approach from disclosure-based regulation — SB 53 mainly required labs to publish safety frameworks and report incidents after the fact — toward enforcement-capable regulation: on-site auditors, a mandated shutdown capability, and incident definitions written to include model behavior, not just outcomes. It arrives with no federal counterpart; Newsom has explicitly framed the order as a template he wants Congress and the White House to adopt nationally. Whether a technically credible shutdown mechanism exists by the panel's first meeting in November matters less than the precedent it sets: a state government is now legislating on the premise that a frontier model might need to be forcibly stopped, an assumption several labs' own safety frameworks already concede on paper but have not been required to make operational.