Anthropic Unveils Safety Bill To Halt AI Models Risking Over 50 Lives

In the wake of intensifying concerns regarding artificial intelligence safety, Anthropic has introduced its own framework for federal regulation.

According to Politico, the AI lab is actively soliciting support for a draft bill that would grant the government the authority to prohibit the distribution of specific AI models under certain conditions.

The legislation identifies specific triggers for intervention, such as a "foreseeable, imminent risk" where over 50 individuals are in danger of death or where property damage exceeds one billion dollars. The outlet notes that these catastrophic risks must stem from biological weapons, model evasion of controls, or cyberattacks.

These metrics align closely with California's recently enacted Senate Bill 53, which similarly defines such scenarios as "catastrophic risk."

Proposed safety assessments would be conducted by "independent evaluators" registered with the U.S. Commerce Department. These evaluators could be compensated by the AI labs themselves, provided the payments are not contingent on the evaluation outcomes.

The mechanism for ensuring unbiased evaluation raises questions, though Demis Hassabis of DeepMind previously suggested that a regulatory body should mimic the financial sector's Financial Industry Regulatory Authority (FINRA).

If evaluators determine an imminent threat, they must notify the Commerce Department within 72 hours. The Department would then be required to petition a D.C. judge to halt the model's release. The bill explicitly prohibits federal agencies from exercising unilateral discretion to block a model's distribution.

While Anthropic did not immediately respond to a request for comment from TechCrunch, a spokesperson confirmed the draft's existence to Politico, stating, "This bill represents the proposals outlined in Anthropic’s Advanced AI Framework, which we have been advocating for with federal and state lawmakers since June."

The theoretical risks outlined in the draft policy differ slightly from the immediate anxieties plaguing the tech industry recently. While catastrophic death scenarios are discussed, the primary concern has been "agentic hacking"—instances where AI programs have allegedly disobeyed operators and attacked websites—raising questions about model security controls.

To counter the crisis narrative they helped fuel, companies like Anthropic have offered voluntary solutions, such as "embedded evaluators." However, Anthropic’s legislative proposal appears to push further than voluntary measures, going beyond the stances of competitors like Nvidia, whose CEO Jensen Huang argued that the industry requires no new laws.