In a recent essay, Dario Amodei, CEO of the AI research firm Anthropic, emphasized the genuine risks associated with advanced artificial intelligence and outlined proposals aimed at ensuring safer AI development. Amodei advocates for a system of embedded evaluators within AI systems as a way to promote coordinated pacing among AI labs in democratic countries, hoping to prevent reckless shortcuts in development. His proposal comes amid ongoing debates about how best to regulate AI and manage competitive dynamics in the industry.
Critics, including figures like Jensen Huang of NVIDIA, have expressed skepticism about the urgency of AI risks highlighted by Amodei. Some argue that Amodei’s emphasis on regulation might serve strategic business interests by creating barriers for competitors, particularly those offering “open-weight” AI models that allow users to remove safety features. However, supporters contend that those competitors, by providing less restricted models, arguably pose greater risks, underscoring the need for regulation.
Amodei acknowledges that relying solely on voluntary adoption of embedded evaluators may fall short. Past efforts, such as DeepMind’s initiative to involve independent evaluators in monitoring AI projects, encountered challenges when disagreements led to dismissals, highlighting limits to self-regulation within the industry. He points to the slow pace of government action, particularly in the United States, where Congress is unlikely to swiftly establish a formal regulatory body overseeing AI labs.
As an alternative, Amodei briefly mentions the idea of a government-backed but industry-financed self-regulatory organization—a concept supported by DeepMind chairman Denis Hassabis. Such a body could provide a framework for cooperation among AI developers while minimizing antitrust concerns, although Amodei and others acknowledge it would not fully resolve the broader challenges.
A major obstacle to meaningful AI pacing identified by Amodei is the competitive pressure exerted by China. The relatively narrow lead U.S. companies hold over Chinese AI firms, measured in months rather than years, limits how much Western developers can slow down safely without ceding technological advantage. Amodei suggests that extending this lead might require tightening export controls on advanced chips and manufacturing equipment, curbing techniques like “distillation” where advanced models are used to train new ones, and enhancing security to prevent intellectual property theft.
Despite these recommendations, efforts to curtail China’s access to key technologies have had mixed results. Since the Biden administration imposed chip-export restrictions in 2022, loopholes have persisted, and Chinese AI models have gained a growing presence in U.S. markets. Western labs, motivated to protect their innovations, have so far struggled to fully prevent technology spillovers.
Recognizing these difficulties, Amodei hints at the potential benefits of engaging China as a partner in AI safety and pacing efforts, despite longstanding geopolitical tensions and human rights concerns. Drawing on parallels from Cold War arms control agreements, he points out that cooperation with China, while challenging, may be essential to managing AI risks globally.
A forthcoming summit between former President Donald Trump and China’s President Xi Jinping, scheduled for September 24, could offer an opportunity for advancing dialogue on AI diplomacy. Given the shared interest of both nations in avoiding the destabilizing effects of superhuman AI, even modest agreements could lay groundwork for further collaboration.
Ultimately, Amodei’s essay underscores that the success of AI safety initiatives will depend not only on technical and regulatory measures but also on diplomatic efforts to bridge divides between democratic and authoritarian powers. Without coordinated action, the difficult path to safely developing powerful AI technologies remains fraught with geopolitical and industry challenges.