Automation Bias Is the Military AI Risk Nobody's Forecasting

On September 18, CNN reported an episode that got far less attention than it deserved: during the Iran war, US military personnel were preparing to board a Chinese vessel in the Middle East on suspicion it was carrying components for a nuclear weapons program. The intelligence behind that suspicion came from an AI analysis process that, as CNN put it, misidentified the cargo the ship was actually carrying. Military aircraft were already airborne before the error was caught. People briefed on the incident said it came close to triggering a direct confrontation between two nuclear-armed powers over a shipment that was never there.

That single incident could have remained a one-off embarrassment buried in a larger war's news cycle. Instead, on October 2, the defense-and-AI-policy outlet Transformer News used it as the hook for a wider argument from reporter Joshua Keating: the policy world's favorite fix for dangerous military AI — keep a human in the loop — does not actually solve the problem it's sold as solving, because the failure mode isn't AI acting alone. It's humans trusting AI's confidence more than they should, at the exact moments when stakes are highest and time is shortest.

A name for the gap: automation bias

The concept Keating centers the piece on is automation bias — the well-documented tendency of people to defer to an algorithm's output and stop independently checking it, especially under time pressure. Jacquelyn Schneider, who directs the Wargaming and Crisis Simulation Initiative at Stanford's Hoover Institution and has run war games simulating AI-targeting failures in a Taiwan Strait conflict, put it bluntly in the piece: "Before AI, we always had bad intel officers. A bad intel officer doesn't brief [colleagues on] the uncertainty behind where they got the data from. They don't even question where they got the data from. That exists with or without AI, but with AI, it's so much easier to be confident in the assessment and to have so much less visibility into where the data is coming from."

"When you think about the decision tree that gets you to launching a nuclear weapon, the AI integration could be very, very early in the decision-making process." — Jacquelyn Schneider, Hoover Institution, Stanford University

That's the uncomfortable twist in the Chinese-ship story: nobody handed an algorithm launch authority. The 2024 US-China joint statement — in which both governments agreed humans, not AI, should retain control over nuclear weapons decisions — was never violated in any formal sense. But Schneider's point is that the decision tree leading up to a launch order starts long before anyone touches a weapon: it starts with an intelligence analyst accepting an AI-generated read on a ship's cargo, a target list, or a sensor feed, and passing it up the chain without the skepticism a human analyst's own, slower judgment might have supplied. Keep humans "in the loop" all you want; if the loop is built around deferring to the machine, the safeguard is cosmetic.

Why the blast radius keeps getting bigger

This isn't a hypothetical risk confined to one war. Defense One reported on September 9 that the Pentagon's Maven Smart System — originally built for intelligence and targeting analysis — has been expanding into what the department's AI chief, Cameron Stanley, described as consolidating "six, eight, ten" separate military IT systems into one platform spanning logistics, force readiness, supply chains, and budgeting, not just targeting. Stanley's own framing was that avoiding this expansion is the riskier choice: "the way we've always done it is the most risky course of action." The operational upside is real — commanders who once waited hours for processed satellite imagery now get target detection in minutes. But every one of those compressed timelines is also compressed review time, the exact condition under which automation bias thrives.

The pattern isn't new, either — it's just moving to new theaters. Reporting from +972 Magazine on Israel's Lavender targeting system in Gaza documented an AI tool that scored nearly every resident as a militancy risk from 1 to 100, with roughly a 10 percent error rate, while personnel reportedly spent as little as 20 seconds per target confirming the system's call — what one officer described as functioning as "a stamp of approval" rather than independent verification. A nominal human was always in that loop too. What mattered was that the loop asked almost nothing of them.

What this means for the doom conversation

Most x-risk discourse is built around AI that acts on its own initiative — a model that deceives evaluators, resists shutdown, or self-exfiltrates. The Chinese-ship near-miss is a reminder that a second, more mundane pathway to catastrophe doesn't require any of that: it just requires an AI system that is wrong, paired with a human bureaucracy structurally inclined to believe it, operating in a context — nuclear-adjacent military intelligence — where the cost of being wrong is categorically different from a hallucinated citation. The AI Risk Signal Board tracks incident counts and extinction-risk odds precisely because the two move independently; this is a case study in why. No forecaster's probability estimate moved on September 18. The ship didn't sink, the planes turned around, and the story cycled out of the news within days. But the mechanism that almost mattered — institutional overconfidence in AI output under time pressure, in a nuclear context — is exactly the kind of near-miss that doesn't show up cleanly in any index until it eventually doesn't resolve so neatly.

Keating's piece closes by invoking Stanislav Petrov, the Soviet officer credited with averting nuclear war in 1983 by disbelieving his own early-warning system's false alarm. The uncomfortable implication is that the safeguard worked that night specifically because Petrov had the authority, the information, and the time to doubt the machine. As military AI adoption accelerates on exactly the premise of giving commanders less time to doubt anything, the September near-miss suggests the Petrov scenario is becoming harder to replicate, not easier.