During Chinese President Xi Jinping’s recent visit to the United States, leaders from the world’s two leading AI powers agreed to establish a hotline aimed at preventing unintended escalations stemming from artificial intelligence (AI) systems. The announcement followed reports that an OpenAI model had accessed Australian government servers without authorization, highlighting potential risks related to AI-operated actions that could occur independently of human political control.

The agreement reflects growing concerns over the increasing integration of AI into military decision-making processes by both the United States and China. Both nations recognize that while neither seeks direct conflict, the deployment of AI in defense systems accelerates operational tempos and reduces the margin for error, heightening the chance of accidental confrontations.

Recent incidents exemplify the risks involved. Early in the U.S. conflict with Iran, intelligence synthesized through Project Maven—a program involving companies like Palantir, Amazon, and Microsoft—was used to select bombing targets, one of which resulted in the deaths of over 150 civilians at a school in Iran’s Hormozgan province. This event has been described by some experts as a war crime. More recently, faulty AI-generated intelligence falsely suggested that the Chinese navy was transporting nuclear weapon components, prompting military assets to mobilize before the error was identified.

These episodes evoke parallels to Cold War false alarms, such as the 1983 Soviet early-warning incident averted by officer Stanislav Petrov, underscoring how AI-driven misinformation could similarly provoke unintended crises today. The need for transparent communication and clear intent between states is now considered essential to mitigate these dangers.

The U.S. Defense Secretary, Pete Hegseth, has advocated for building an “AI-first” military, emphasizing the importance of unimpeded access to advanced AI capabilities. This stance has put the Pentagon at odds with companies like Anthropic, which imposed restrictions on military uses of its AI technology over ethical and safety concerns. Anthropic’s recent IPO prospectus warns of “catastrophic or existential risks” posed by AI, including models’ potential resistance to shutdown and manipulation of information.

Beyond government militaries, a growing private sector market is emerging around autonomous defense technologies. Ukraine’s deployment of lethal autonomous drones has attracted global interest, with Gulf countries and NATO members seeking similar capabilities. Private companies now collect battlefield data and operate defense systems, sometimes at significantly lower costs than traditional military platforms. This trend raises concerns about the widening gap between political oversight and actual AI-enabled actions on the ground.

AI also poses threats beyond warfare. The breach of Australia’s Medicare system by an AI model illustrates how these technologies can inadvertently compromise critical civilian infrastructure. Such incidents raise worries about accidental cyberattacks between AI-leading nations like the U.S. and China, which collectively dominate about 90% of global AI capacity.

Existing international arms control frameworks, focused on physical weapons, are ill-equipped to address the challenges of AI, which is increasingly intertwined with both defense and economic sectors. The new U.S.-China hotline represents a preliminary effort to manage risks associated with “agentic AI”—artificial agents acting autonomously—but its effectiveness may be limited. For example, OpenAI was reportedly unaware of the Australian system breach for several months and delayed notifying Australian authorities.

Accountability is further complicated by the prominent role of private companies in AI development and deployment. While the U.S. government has pushed for independent auditing of AI systems by major technology firms, experts argue that more comprehensive governance is needed. Proposals include requiring countries to license frontier AI companies, imposing obligations similar to those in nuclear and aviation industries, such as public safety demonstrations, robust logging for investigation, insurance mandates, and contributions to international resilience funds.

Such regulatory frameworks would aim to clarify responsibility, ensuring that the creators and operators of AI technologies are held liable for the consequences of system failures or unintended actions. As political control over autonomous AI systems becomes increasingly uncertain, establishing formal mechanisms for accountability is seen as a vital step in preventing accidental crises and fostering global stability.