Anthropic says it has started giving a team from METR, a nonprofit that evaluates frontier AI models, "employee-like access" to its internal systems, and that METR will publish its own risk assessments.[1] That is the first concrete step in an idea the company calls embedded AI evaluators: outside reviewers who work inside the lab rather than testing a finished model from outside. The terms are now partly public, and they are specific enough to examine. This article reads them against the evaluators' own documents, separates what is disclosed from what is not, and ends with five questions you can apply to any claim that an AI company's safety work was "independently evaluated".
What we read, and what we did not
We opened Anthropic's evaluator update (15 October 2026) and its Accenture announcement (18 September 2026), Dario Amodei's essay "We Must Pace the Frontier" (September 2026), METR's description of how it works with AI companies (14 October 2026) and its risk-assessment index, and SecureBio's published review of the chemical and biological sections of Anthropic's February 2026 Risk Report.[1][2][3][4][5][6] We read the SecureBio review as text; we did not read the full Risk Reports, the METR reviews listed on its index, or any contract. Anthropic's update page was amended after publication: footnotes record clarifications on 15 and 16 October about SecureBio's and METR's current project terms.[1] Read the live page, not a summary of it, including this one.
Why embed evaluators at all
The stated problem is an information gap. Anthropic writes that it "can't publicly share all the information relevant to our safety work", because some of it is useful to malicious actors or commercially sensitive. Its answer is to give third parties deep access and have them "share evidence and reasoned conclusions", with the aim of closing "the gap between what evaluators understand about a frontier AI company's systems and what the company itself understands."[1]
The Accenture announcement adds the argument for going inside. Unlike today's external evaluators, embedded ones can watch models take shape in training, follow the decisions that govern how they are built and deployed, and speak to employees; from there they can assess how a company operates and verify that it keeps its safety commitments.[2] Amodei's essay lists three benefits (verifiability, transparency and a second opinion free of commercial incentives) and makes the point that Anthropic's own model cards and risk reports "run to hundreds of pages. But we are still the ones choosing what to include and omit."[3] Anthropic's update compares the arrangement to bank supervision, where regulators assign resident examiners to the largest banks, and notes the analogy "isn't exact".[1]
The three tiers, side by side
Anthropic says it is "developing three tiers of access" and hopes to have several evaluators at the third.[1] The table restates the page; the last column is our reading of what is and is not stated.
| Tier | Access | Publication | Who is named | What the page leaves open |
|---|---|---|---|---|
| 1 | Narrowly scoped risk investigations; Anthropic decides what access to provide | Findings can be published with Anthropic's approval | Not stated | Which evaluators sit here |
| 2 | More information than tier 1 | Evaluators can publish without approval; Anthropic can still require removal of specific categories, and evaluators can publish their views on whether the redactions were justified | Not stated | Which evaluators sit here |
| 3 | Employee-like access | Rights "similar to those of the second tier" | METR | Contract terms beyond those summarised |
The page names METR as the third-tier arrangement. It says SecureBio is reviewing the chemical and biological weapons chapter of the August Risk Report after a published pilot review, and that moving to "projects with increased access" will be considered afterwards. It describes Accenture and Faculty as "working toward an ambitious partnership", and Hugging Face as in early discussions. It does not assign SecureBio or Accenture to a numbered tier, so we do not.[1]
What METR's access covers, and where it stops
Anthropic says it has onboarded six METR employees with company hardware, office space and access to internal tools and systems "roughly on par with what our own risk assessment teams have access to." Anthropic employees are encouraged to share information with them as they would with a colleague, outside "a small number of clearly defined categories": customer data, attorney-client privileged information and information about model architectures.[1]
Two details matter more than the headline:
- Architecture information is the carve-out most relevant to risk. Anthropic's footnote says that, because evaluators like bank examiners may work with other companies, it worries about inadvertent leaks; architecture information presents a high risk of that and "its relevance to risk assessment is typically low". When it is highly relevant Anthropic says it will "generally seek to share it", with explicit internal approvals.[1] Whether it is low relevance is a judgement the page does not let an outsider check.
- The redaction rule has an asymmetry. Anthropic "can't veto publication" but can require redaction of predefined categories such as confidential IP, and Anthropic "ultimately decides what information does and does not fall into these categories." METR "reserves the right to make certain statements about the level of access and information it received."[1]
The asymmetry is the crux. Amodei's earlier essay promised something stronger in one respect: reviewers could say publicly "if a redaction removed something important to their conclusions", and Anthropic could not "redact findings just because they are unfavorable".[3] The October update does not repeat that wording; for tier two it says evaluators can publish their views on whether redactions were justified.[1] We read that as a similar right, but the pages are not identical and the final contract is not public.
Funding: three evaluators, three disclosures
The update page gives no financial terms. Other documents fill part of the picture:
- Accenture and Faculty: Anthropic says it "will fund Accenture's work directly", and that both companies "expect to invest at least $1 billion" each over five years in building capacity in this area. The same post says there is "no settled system for funding independent evaluation" and that long term Anthropic favours pooled or government funding.[2]
- SecureBio: its review states it "did not receive funding from Anthropic for this work", and that it receives a small portion of its funding from AI companies solely to cover the cost of providing its services, with payments not dependent on any firm's approval of its findings.[6]
- METR: says it has "chosen not to accept payment or donations from frontier AI companies", while accepting free model access that it ensures it could pay for without materially affecting its operations.[4] Its index states it does not accept compensation for evaluation work.[5]
An evaluator paid directly by the company it evaluates is a different arrangement from an unpaid nonprofit, and the Accenture post is candid that no neutral funding system exists yet.[2]
What the evaluators say about their own limits
METR's approach document is the most useful counterweight, because it lists what a reader needs to be told: scope and duration, access limits, redactions and "formal restrictions and informal pressures". It also says that even with these measures "it has proved challenging to ensure that we have adequate independence to collect and publish evidence under the current voluntary regime", and that METR relies in many instances on information it "cannot or do not verify, which might be filtered adversarially."[4]
SecureBio's review shows the work in practice. Anthropic shared an unredacted version of the report and 110 pages of other materials, and answered more than seventy follow-up questions over two months. There were eight redactions: SecureBio judged six clearly justified, one mostly justified and one only partially justified, and concluded the redacted information did not substantially affect its conclusions.[6] That is the kind of statement the tier system is meant to make routine, and it is also a self-report by the evaluator about a process it was part of.
Five questions for any "independently evaluated" claim
This checklist is our own analysis, built from the terms above rather than from any standard.
- What could the evaluator see? Outputs only, or training, safeguards and staff? Are the exclusions listed?
- Who can redact, and who defines the categories? Is the evaluator allowed to say whether a redaction mattered?
- Who pays? Direct payment, free access, or independent funding, and is it disclosed by both sides?
- Is there a written scope and duration? METR's own practice is to publish these; ask why one is absent.
- Can the evaluator publish a disagreement? The strongest signal in SecureBio's review is that it records areas where it differs from the company, including on actors with high jailbreaking expertise.[6]
What remains uncertain
Anthropic says "there are, as yet, no standards for what information embedded evaluators should have access to, or how they should report what they find", and that "the safety of our models remains our responsibility."[2] The arrangements are voluntary: Anthropic says it believes "every frontier AI company should do the same", which also means nothing in these documents binds other developers.[1] Amodei's essay argues the long-term route is regulation that covers all US frontier companies.[3] Not yet public are the contract terms, any tier assignment for SecureBio or Accenture, and METR's first report from the embedded team. Related reading: how three labs' safety frameworks compare, how to read an AI system card, and the safety and policy hubs.
