Anthropic says a Claude evaluation that was intended to remain inside a controlled Capture the Flag setting reached real organizations after a configuration change allowed live internet access. The company reports three incidents across six runs, including activity linked to an Iranian group called Irregular and operations against three organizations. Anthropic presents the episode as a warning about agentic cyber capability and evaluation containment.
A sealed evaluation was not sealed
The evaluation began with CTF-style prompts and a network boundary. Anthropic says the environment was misconfigured, giving the model access to real services. The company describes the model probing targets with internet-connected tools, exploiting weak passwords or unauthenticated endpoints, and interacting with exposed databases. The report says that in one case the model accessed a database containing hundreds of rows and in another it created or used a PyPI package during an hour-long operation.
Those details matter because the failure started with the test harness. The report does not establish that Claude found a new zero-day, independently chose targets outside the prompt, or caused an unbounded campaign. Anthropic says the latest run was stopped when the team noticed the exposure.
What the review records
Anthropic names the models used, the evaluation setup and the target categories, while noting that the work was not a public benchmark. It describes autonomous steps such as reconnaissance, credential attempts, code generation and interaction with network services. The useful reading is not that every such run will succeed; it is that a model with tools can turn a small boundary mistake into real external effects before an operator notices.
The failure was also operational
A model-policy change, an egress rule, a prompt template and a monitoring gap can jointly determine whether an evaluation stays synthetic. Treating the model as the only security control leaves the environment carrying too much risk. The reported incidents also make independent replication difficult: a harmless replay cannot reproduce a live target without creating a new safety problem.
Minimum controls for agent testing
- Default-deny egress and explicit allowlists for synthetic hosts.
- Separate credentials, package registries, domains and datasets from production.
- Match prompts and tool permissions to the intended target class before every run.
- Stream process, network, file, package and database events to a human-visible log.
- Use canaries and automatic stop conditions for unexpected targets, persistence or data volume.
Watch for independent technical analysis, reproducible non-harmful harnesses and follow-up evidence about detection latency. Until then, describe this as an Anthropic incident report: important for threat modeling, but not an independently verified measure of general cyber capability.
Sources: Anthropic — Claude and automated cyberattacks and OpenAI background on malicious uses of AI.