How can a model developer find weaknesses in a safety filter before someone else does? Anthropic's answer is Anvil, an internal system that uses Claude to repeatedly test selected safeguards. In a report dated November 12, the company says Anvil found stronger failures than human red-teamers in several of its evaluations, often in hours or days rather than weeks. Those are Anthropic's own results on its own systems; they are not an independent comparison of red-teaming tools or a measure of how often real users can bypass safeguards.
What is Anvil testing?
Anthropic describes Anvil as a set of parallel investigator sessions. The agents try to make progress on tasks that the target model's safeguards are intended to block, then use the results to identify gaps in the classifiers. Anthropic says the system builds on earlier investigator-agent work. Its report covers three areas: biological misuse safeguards, cybersecurity safeguards and classifiers intended to prevent extraction of hidden model reasoning.
The important distinction is between the model acting as an attacker inside an evaluation and a normal product interaction. Anthropic says its strongest runs rely on internal frontier models with safeguards disabled for the attacker, privileged access to classifier scores and task-specific setup. The company says those conditions make it difficult for an outside actor to reproduce the system today. It also warns that improving open models could change that balance over time.
What did the comparisons show?
For one biological-safeguard evaluation, Anthropic compared Anvil with its bug-bounty results across eight questions and three generations of production classifiers. The report says Anvil matched or exceeded the strongest human submission in all 24 question-classifier comparisons, winning 23, and found its strongest results in 24 to 48 hours. Anthropic's public roadmap says this work completed an internal goal ahead of its January 2027 target. The score and the choice of comparison are Anthropic's; the report does not establish how an external evaluator would rank the same cases.
The report also describes selected cybersecurity and anti-distillation tests. Anthropic says Anvil found more successful failures than contracted experts on the tasks it compared, and that some of its classifier tests helped the company improve later defenses. It reports a drop in the share of Anvil attacks missed by successive anti-distillation classifiers, alongside a decline in a particular reasoning-extraction pattern seen in its own traffic. These measures relate to those specific safeguards and evaluations; they should not be read as a general security score for Claude or for AI systems overall.
Why is automated red-teaming useful—and risky?
A human review can be valuable but slow, especially when a team needs to compare many versions of a safeguard. An automated evaluator can run repeated tests, help surface cases that people missed and turn those cases into a regression set for the next version. Anthropic says it uses Anvil before deploying new safeguards and to refine defenses. In principle, this makes evaluation part of the release process rather than a one-time approval step.
The same capability can intensify the contest between attackers and defenders. Anthropic says Anvil improves with the model powering it, while also stressing that its current setup depends on privileged access and internal information. Its report therefore supports two narrower conclusions: automated systems can help a developer probe its own safeguards at scale, and access to stronger models and evaluation evidence can create an advantage for the defender. It does not show that this advantage will persist as tools become more widely available.
What should model builders take from the report?
The useful takeaway is about the evaluation loop, not a recipe for bypassing a filter. Test a safeguard against diverse, capable attempts before release; preserve enough evaluation detail to compare versions; use findings to improve the system; and keep rechecking after changes. The results should be paired with limits on the test environment, human review and a clear account of what the benchmark can and cannot represent. A model that passes one red-team suite is not thereby proven safe in every product, task or deployment.
Anthropic's report is a vendor-authored account of an internal capability, and many details of its production classifiers are not public. VIBPS has not reproduced Anvil's evaluations or independently measured its claims. For software makers, the broader point is that AI-assisted development and security depend on evidence and review around the model, not only on the model's apparent behavior in a demo.
How OpenAI describes stopping a research-model workload
How to protect an app while an AI coding agent works
VIBPS is a social network for software: makers can publish what they build and give people a way to follow their work. Research into how software and AI systems are evaluated matters because those systems increasingly shape the tools makers use.