The Automation Trap

The finding had just been put on the table, and the table was absorbing it. Put a large language model into a nuclear escalation scenario, run it enough times, and it raises the nuclear option in roughly nine runs out of ten and recommends actually using a warhead in roughly seven. Around me sat people who had spent their working lives on disarmament.

Then I told them I had seen the thing they were theorising about. Not a proposal, not a pilot. A G7 navy with an artificial-intelligence agent aboard its ships, ranking the incoming aircraft, drones, missiles and torpedoes and presenting the commander with the order in which to engage them, on a single screen. It is at sea now. Someone asked me what I would do in that chair.

The honest answer is that nobody can answer that question, and that is this episode. This is not Top Gun. The commander has seconds, not even the ten minutes a nuclear decision is supposed to allow. No human being adjudicates a ranked threat list in the time available. He confirms the recommendation, or his crew of two thousand finds out what the torpedo was for.

We have a name for this arrangement. Human in the loop. It lets governments adopt systems they do not understand, and sleep. But the safeguard assumes two things that a compressed engagement removes. It assumes the human has time, and it assumes the human has information the machine does not. When neither holds, the human's presence does something worse than nothing. It launders the decision. Automation bias is well documented: faced with a confident recommendation and a clock, people overwhelmingly confirm. The loop becomes a signature, not a judgement.

I defended that commander in Rome and I would again. Given the system in front of him, confirming it was the right call; refusing the tool and hoping is not an ethic, it is a way of losing a ship. The trap is not in the chair. It is upstream, in the decision to put that chair in front of that screen and call the result human control.

So the distinction that matters is not human versus machine. It is logistics versus targeting. In logistics, the unglamorous bulk of what a navy does, the machine is a tool, and we should use it. In targeting, the machine is being asked to act as a moral agent, and the encyclical names that category error exactly: "moral judgment cannot be reduced to calculation, for it involves conscience, personal responsibility and the recognition of the other as a person. Therefore, it is not permissible to entrust lethal or otherwise irreversible decisions to artificial systems. No algorithm can make war morally acceptable."

The consequence for anyone trying to regulate this is uncomfortable. By the time the click exists, every meaningful decision has already been made: which data the system learned from, what it was taught to rank, how it presents the ranking, how long it gives the human. Regulate the pipeline, or you have regulated nothing. The Rome Declaration, adopted at the Campidoglio in July, calls in its third principle for a treaty prohibiting the reckless integration of AI into nuclear command, control and launch systems "so that meaningful human control is maintained." Cardinal Reina, at the signing, put it without the lawyers: no machine, no algorithm, no autonomous system can be placed at the centre of decisions on which the survival of humanity depends. The encyclical goes one step further and gives the standard that should replace the slogan: "effective, self-aware and responsible human control." Self-aware is the word. A human who confirms is not in control. A human who knows what the machine cannot know, and has the time to act on it, is.

Here is where I parted company with the room. One of the disarmament veterans put it to me that AI takes the annual probability of nuclear use from something like one per cent to something like three. Perhaps. But AI is not the principal preoccupation. The existence of the weapon is. If the decision window goes from twenty-three minutes to ten, the end result is the same. The world has come down from a Cold War peak of some seventy thousand warheads to roughly twelve thousand, and the Doomsday Clock stands at eighty-five seconds to midnight anyway. AI is an accelerant on a hazard we chose to keep. That does not make the treaty less urgent. It makes it more so, because an accelerant is the one thing you can still remove from a fire you have decided to live beside.

The compression of decision time is the actual threat, and it cannot be reversed by putting a person in front of the screen. It can only be reversed by deciding, in advance and in writing, which decisions may never be compressed. The Declaration's final principle asks nuclear states for exactly this: to "extend decision time" and move away from launch-on-warning. The encyclical says it in one line: "speed and efficiency should never be the supreme motivating force for the irreversible decisions made in the context of war."

Previously: Beyond the Hegemon. Next Tuesday: The Un-Glamorous Revolution.