Modern warfare is undergoing a quiet but fundamental transition. In theaters of conflict from the Middle East to the Pacific, artificial intelligence has moved beyond the periphery of data analysis to become an active, real-time participant. From generating targets to coordinating the intricate maneuvers of drone swarms, AI systems are now embedded in the kinetic chain — the sequence of decisions that begins with identifying a threat and ends with the application of lethal force. This shift has sparked a legal and ethical tug-of-war, exemplified by the friction between the Pentagon and developers like Anthropic over the boundaries of military automation.
To mitigate the risks of these autonomous systems, the Pentagon relies on the doctrine of the "human in the loop." The framework suggests that as long as a person oversees the decision-making process, the system remains accountable, nuanced, and safe from the whims of a rogue algorithm. However, this oversight may be more performative than protective. As these systems grow in complexity, the human supervisor often becomes a mere rubber stamp for processes they cannot truly comprehend.
The black box on the battlefield
The fundamental flaw in current military guidelines is the assumption of transparency. State-of-the-art AI remains a "black box" — an opaque architecture where inputs lead to outputs through logic that even its creators cannot fully map. Deep neural networks, the class of models underpinning most modern AI, derive their power precisely from the density and nonlinearity of their internal representations. No single layer of weights encodes a legible rationale. When an operator acts on an AI-generated target, they are not exercising informed judgment; they are trusting a system whose internal reasoning is effectively unknowable.
This is not a new epistemological puzzle. The challenge of interpretability — the subfield of machine learning devoted to making model decisions explicable — has occupied researchers for over a decade. Progress has been made in narrow domains: saliency maps can highlight which pixels influenced an image classifier, and attention weights can suggest which tokens a language model prioritized. But these tools offer post hoc approximations, not ground truth. In the high-stakes environment of the battlefield, the gap between approximate explanation and genuine understanding carries consequences measured in human lives.
The problem compounds at speed. Autonomous drone coordination operates on timescales that compress the decision cycle from minutes to seconds. A human supervisor reviewing an AI-generated strike recommendation may have neither the time nor the technical literacy to interrogate the model's reasoning before a window of action closes. The doctrine of meaningful human control assumes a tempo of deliberation that modern autonomous systems are explicitly designed to eliminate.
Accountability without comprehension
International humanitarian law rests on the principle that someone can be held responsible for the decision to use force. The human-in-the-loop framework is, in part, a legal architecture: it assigns a point of accountability. But accountability presupposes agency, and agency presupposes understanding. If the operator cannot reconstruct why the system flagged a particular target — cannot distinguish a valid military objective from a pattern-matching artifact — the chain of responsibility becomes a legal fiction.
The tension is not unique to the military domain. In finance, in healthcare, in criminal sentencing, automated systems have outpaced the capacity of their human overseers to audit them in real time. What distinguishes the military context is the irreversibility of the outcome and the absence of a corrective feedback loop. A misclassified loan application can be reversed; a misclassified target cannot.
Historical precedent offers a cautionary frame. The introduction of radar and electronic warfare in the mid-twentieth century similarly outstripped the cognitive bandwidth of human operators, leading to incidents of friendly fire and misidentification — the 1988 downing of Iran Air Flight 655 by the USS Vincennes being among the most studied. The difference today is one of degree: AI systems are not merely presenting data faster than humans can process it; they are making inferential leaps that humans cannot replicate even given unlimited time.
The Pentagon's challenge, then, is not simply one of policy design or interface engineering. It is a confrontation with a structural asymmetry: the capabilities that make AI valuable in warfare — speed, pattern recognition across vast datasets, coordination beyond human cognitive limits — are the same capabilities that render meaningful oversight difficult. Slowing the system down to accommodate human review negates its tactical advantage. Leaving it at full speed reduces the human role to a ceremonial checkpoint.
Whether the doctrine of human-in-the-loop governance can evolve fast enough to match the systems it purports to govern, or whether it will calcify into a convenient legal shield that satisfies no one — least of all the operators asked to bear responsibility for decisions they did not truly make — remains the central unresolved tension in the militarization of artificial intelligence.
With reporting from MIT Technology Review.
Source · MIT Technology Review
