Moving evaluation behind the public interface
Anthropic has begun giving an independent evaluator access closer to that of its internal risk teams. Announced on 15 October, the arrangement embeds six employees from the nonprofit METR with company hardware, office space and access to internal tools and systems. The aim is to narrow the information gap between a frontier-model developer and outsiders asked to assess its risks.
External testing usually observes a model through an API, selected documentation and structured briefings. That can reveal important behaviour but not always the decisions, incidents or safeguards surrounding development. Anthropic says employees are encouraged to share information with the embedded team as they would with a colleague working on alignment risk, outside defined exclusions.
Publication independence is explicit
METR will publish analysis of Anthropic risk levels, incidents and practices. Anthropic cannot veto publication. It may require redaction only within narrow categories established in advance, including customer data, legally privileged information and details of model architecture. METR can publicly characterise the redactions and say whether it considers them justified without disclosing the protected content.
That structure is designed to avoid a review that becomes private consulting with a public summary controlled by the company. Publication authority gives the evaluator room to present unfavourable conclusions. Predefined exclusions recognise that full public access would expose customer information, security-sensitive designs and legal material that no responsible audit could simply release.
Access remains contained
The arrangement applies to a small METR team working under enforceable agreements and Anthropic security standards. Other METR staff are siloed from the embedded work, including researchers involved in separate projects with Anthropic. The company says this is not a route for METR or other organisations to obtain customer and partner information.
Containment is important because deeper evaluation access creates its own security risk. Evaluators need enough evidence to reach independent conclusions, but access should be logged, role-based and limited to the work. The credibility of the exercise depends on both independence and disciplined handling of information that could harm users if disclosed.
Several evaluators cover different risks
Anthropic is also working with SecureBio and Accenture and is in early discussions with other organisations, including Hugging Face. Different evaluators can bring expertise in biological risk, enterprise controls, model capability and open technical ecosystems. A single reviewer is unlikely to cover every risk created by increasingly capable models.
The company says it has worked with METR on reviews of Risk Reports over the past year. The new arrangement goes further by allowing evaluation of the systems and practices that produce those reports. Public findings should make clear what evidence was inspected, which models and controls were in scope and where access limitations prevent a strong conclusion.
A test for embedded oversight
Employee-like access does not make an evaluator an employee, and proximity can create familiarity as well as insight. METR will need methods for maintaining independence, rotating reviewers and documenting disagreements. Anthropic should respond publicly to findings without turning the evaluator into a substitute for regulators, academic scrutiny or internal accountability.
The model may offer a practical middle ground between black-box testing and unrestricted disclosure. Its success will be measured by the specificity of published assessments, evidence that difficult issues can be reported and changes made after findings. If those conditions hold, embedded independent evaluation could give the public a clearer view of frontier risk than company-authored assurances alone.
What to watch next
Comparable reporting over time would make the arrangement more valuable. If METR explains methodology, access boundaries and unresolved questions in each assessment, readers can distinguish a change in model risk from a change in what the evaluator was allowed to see. Anthropic can also publish remediation milestones while leaving the evaluator free to judge whether those changes address the original concern.
Practical implications
The embedded team should have access early enough to influence risk decisions, not only inspect a release after choices are fixed. It also needs the ability to follow an issue across model versions and verify whether mitigation changed behaviour. A snapshot assessment can miss risks that emerge through deployment scale, tool use or a later system update.
Conflicts of interest should be disclosed clearly. Evaluators may receive funding, access or operational support from the company they assess, and those resources can be necessary for deep work. Public descriptions of contracts, publication rights and prohibited interference help readers judge whether independence is meaningful rather than assumed.
Other frontier developers will watch whether this approach produces information that conventional safety reports did not. If it does, the design could become a reference for auditors and governments. If published work stays broad despite deep access, the arrangement may improve private understanding without closing the public evidence gap it was created to address. The practical test is whether difficult findings remain publishable when they arrive close to a commercial launch decision.