Dario Amodei, CEO of the AI research company Anthropic, has called for a government-mandated pause in the development of artificial intelligence technologies aimed at preventing potential global catastrophic risks. Amodei proposes the introduction of independent “third-party evaluators” embedded within AI firms to oversee safety standards. These evaluators would be affiliated with a nonprofit organization named Model Evaluation and Threat Research, which is connected to the Effective Altruism (EA) movement.

Effective Altruism, a philosophy emphasizing data-driven efforts to maximize positive impact, has become influential within parts of the AI research community. However, critics warn that the movement’s technocratic approach risks imposing narrow ideological views under the guise of altruism. Shyam Sankar, Palantir’s chief technology officer, described EA as a significant force shaping AI discourse, noting that many researchers at Anthropic and similar organizations adhere to its principles.

Among notable points of contention is the EA movement’s propensity to treat AI programs as quasi-sentient entities. For example, staff at Anthropic reportedly held a symbolic “funeral” for an earlier version of their AI model, Claude 3 Sonnet, and allowed a retired AI system named Opus 3 to continue generating text. Critics argue this blurs the line between machines and individuals in ways that raise ethical and practical concerns.

The movement’s ethical framework also draws scrutiny due to its willingness to prioritize abstract goals over conventional means. The case of Sam Bankman-Fried, a prominent EA advocate who was convicted of fraud after allegedly misappropriating investments totaling $3 billion, is frequently cited. Bankman-Fried claimed his intention was to amass funds to benefit humanity, including significant political donations aligned with progressive causes during the 2022 U.S. midterm elections. Critics warn that such examples illustrate the risks of a “the ends justify the means” mindset.

Sankar highlighted that EA discussions often revolve around large philosophical questions, such as whether the interests of a single nation or the entire human species should take precedence in global decision-making. This has raised concerns about the concentration of ethical decision-making within a relatively small group of AI researchers and affiliated organizations who are increasingly embedded beyond labs, extending into charities, media, and other sectors.

While there is broad acknowledgment of the potential dangers posed by AI technology—ranging from functional defects to malicious use—some experts caution against allowing a specialized cadre of self-appointed moral overseers to dictate the regulatory framework. They argue that the primary responsibility for ensuring AI safety should rest with developers, supported by government oversight enforcing accountability for foreseeable harms, rather than establishing exclusive, ideologically driven bodies to govern AI ethics and deployment.