Source Body Text

As tech leaders warn about the threats of artificial intelligence, most conjure up images of AI agents running amok, such as hacking systems used to run our critical infrastructure, banks or even militaries. Yet one serious danger is mentioned less frequently – the threat posed by AI–empowered lethal weapons. Fully autonomous versions of these weapons are colloquially known as killer robots. Israel’s conduct in Gaza shows that, in an important respect, such AI weapon systems are already here. An algorithm has been deciding who lives and who dies. For more than a decade, governments have been meeting in Geneva to stop the development of autonomous AI weapons. In early September, 128 governments met to approve a report that could serve as the foundation for a binding treaty. But a collection of lawyers sent by Donald Trump and Vladimir Putin teamed up to water it down. Notably, they weakened a provision that requires humans to review military targets developed by AI before a strike. Fully autonomous weapons remain the focus; the use of AI to determine targets for conventional weapons is not. At risk is the longstanding quest to maintain a “human in the loop” – that is, meaningful or effective “human control” of AI-empowered weaponry. For many people, talk of killer robots conjures up images of Arnold Schwarzenegger’s The Terminator. I have often thought about the problem in the context of the Tahrir Square uprising at the height of the Arab spring that led to the ouster of the Egyptian president Hosni Mubarak. Mubarak’s fate was sealed when the Egyptian army made clear it would refuse to fire on Egyptian protesters. But killer robots would have had no such qualms. If Mubarak had been able to deploy them, he might have stayed in power. The negotiators in Geneva are not even addressing this problem. The Trump-Putin lawyers made sure that the agreed report referred only to war, not repression. Today, amid the drone warfare in eastern Ukraine, Russian and Ukrainian troops are experimenting with fully autonomous weapons to overcome jamming of the signals used to control drones. We can expect these nascent deployments to expand. Yet the “killer robot” metaphor is deceptive. If we look at targeting decisions more expansively, AI has already taken over. In Gaza, the Israeli military is using AI to target Palestinians. Israel has long monitored phone communications in that territory. After the Hamas attack of 7 October 2023, Israel used an AI system called “Lavender” to uncover patterns that it equated with the phone of a Hamas operative. People whose phones AI determined met these criteria could be killed. Because it was easier to target suspects at home, a separate automated system called “Where’s Daddy?” tracked them there, meaning that their families were often killed too. There was no pretense that Hamas was using its operatives’ homes for military purposes. It was simply easier for the AI system to find suspected Hamas fighters where they slept. The families were the collateral damage. Theoretically, a human remained in the loop because an AI-picked target was routed through intelligence officers who approved it and sent it to others who launched the attack. In fact, this human role was nominal. In the past, Israeli targeting decisions were subject to significant review. But in Israel’s killing spree that followed the Hamas attack, the intense demand for targets took precedence over such review. Israeli intelligence officers who received the AI selections routinely passed them to the attackers without second guessing. As one officer described: “I would invest 20 seconds for each target at this stage, and do dozens of them every day. I had zero added-value as a human, apart from being a stamp of approval. It saved a lot of time.” From computer to killing, the human judgment exercised was perfunctory – a “rubber stamp”. In essence, an algorithm played God. One Israeli officer suggested the practice might be justified because AI makes better decisions than humans do. It is dispassionate, while people’s judgment can be clouded, such as by the animosity toward all Palestinians that some Israelis felt after the Hamas attack. As one Israeli soldier explained: “I have much more trust in a statistical mechanism than a soldier who lost a friend two days ago. Everyone there, including me, lost people on October 7. The machine did it coldly. And that made it easier.” Yet AI algorithms are inevitably affected by human input. How much evidence is required to identify someone as a Hamas fighter? What margin of error is tolerated? The answer to such questions depends on the judgment of the AI programmers and their commanders – ordinary Israelis who might have devalued Palestinian civilian life in the fury that followed the Hamas attack. Even the Israeli military conceded that about 10% of the human targets slated for assassination were not members of the Hamas military wing at all. The Israeli president, Isaac Herzog, epitomized the heedless attitude toward Palestinian civilians when he said: “It is an entire nation out there that is responsible. It’s not true, this rhetoric about civilians were not aware, not involved – it’s absolutely not true. They could’ve risen up, fought against that evil regime.” That same contempt contributed to the Israeli military’s shocking willingness to tolerate civilian deaths. The Israeli military permitted an anticipated 20 dead for the killing of a mere suspected low-level Hamas fighter. Under the rule of proportionality, international humanitarian law prohibits a wartime attack if it “may be expected to cause incidental loss of civilian life … which would be excessive in relation to the concrete and direct military advantage anticipated”. Although international law doesn’t specify the number of civilian casualties that are acceptable in a given situation, virtually no legal expert considers Israel’s shocking 20-to-one ratio to be lawful. These are war crimes. They reflect an appalling devaluing of Palestinian civilian life. Evoking for me Hannah Arendt’s “banality of evil”, ordinary soldiers became complicit in extraordinary cruelty by trusting what they call the “system”. The essence of that system remained people – the generals who ordered its creation, the military lawyers who excused the unjustifiable, the programmers who tweaked the algorithm, the intelligence officers who passed on the computer-generated targets, and the soldiers who launched the attack. But part of what made it easier to trust the system was the pseudo-scientific precision of AI. We are all familiar with the breezy, authoritative tone of today’s chatbots. When AI is used for targeting, it brings a similar feigned omniscience to killing. When inevitably flawed human beings make decisions, they can expect others to challenge them. But AI makes it easier for us to defer to its judgments. That deference risks magnifying the human biases that inevitably infect any AI system. As we seek to curb killer robots, we should also remain attentive to AI that can lull us into complacency about other targeting. The International Committee of the Red Cross calls it “automation bias”. To have meaningful human control of AI, we need not just nominal endorsement of its decisions. We need intense questioning of its product, and accountability for commanders when junior officers fall short. Only that skepticism has any hope of ensuring that soldiers who deploy AI remember that real human beings are at the end of the kill chain they enable. Even in war, people should never be snuffed out by the mechanical application of an algorithm. Kenneth Roth is a Guardian US columnist, a senior fellow at Yale University, and a former executive director of Human Rights Watch. He is the author of Righting Wrongs: Three Decades on the Front Lines Battling Abusive Governments