Independent embedded evaluators at frontier AI labs
Loading…
For each laboratory and date, will there be clear public evidence that the laboratory has put its September 2026 commitment to independent embedded evaluators into substantive practice?
Resolve YES when a new or materially expanded, continuing engagement is operating: independent evaluators have meaningful access to internal systems, staff and AI development or safety practices, and meaningful independence to assess and communicate their findings broadly in line with the laboratory’s commitment. The substance of the arrangement matters; there is no minimum number of days, requirement to publish the contract, or checklist of office equipment.
A future promise, an ordinary external model evaluation or a prior bounded exercise alone does not count. Credible confirmation from the laboratory, evaluator, published findings or reliable reporting is sufficient; both parties need not issue announcements. Resolve NO for a date if qualifying public evidence has not appeared by its end, even if later evidence reveals an earlier private start. A qualifying engagement subsequently ending does not undo earlier YES outcomes.
The two categories are Anthropic and OpenAI. The reference commitments are Dario Amodei’s September 2026 proposal and Sam Altman’s endorsement. Narrow confidentiality or security redactions are compatible with the commitment; routine company control over unfavorable conclusions is not.
Bars include events up to their close, 23:59:59 UTC: “By end of Sep 2026” includes everything up to 30 Sep 2026, 23:59:59 UTC.
The starting probabilities of this market were calculated from the work needed to turn the two companies’ commitments into an ongoing evaluator programme: choose a team, extend its access, agree reporting rights and begin work.
Both labs have already hosted evaluators inside their organisations. METR’s Anthropic exercise gave a researcher internal systems access and staff support for three weeks. Its OpenAI investigation involved several visits, interviews and access to internal records. These relationships make an expansion of existing arrangements a plausible route.
The main uncertainty is how much work remains on staffing, access and the right to communicate findings. The estimates use three possibilities:
| Route | Weight | Timing from the commitment |
|---|---|---|
| Extend an existing relationship; most arrangements are ready | 55% | Typically around six weeks |
| Substantial work on staffing, permissions or reporting terms | 35% | Typically around five months, with a wide range |
| The commitment stalls or settles into an arrangement that falls short | 10% | Remains unresolved |
These weights and times are judgments based on the work involved. Public accounts of earlier exercises give useful evidence about access and feasibility, but leave the time spent negotiating arrangements unclear. The model allows considerable variation around each timeline.
Anthropic receives a timing advantage because it developed the proposal and described concrete access and reporting arrangements. That suggests more preparation before the announcement. OpenAI endorsed it the same day and can build on its own evaluator relationships.
A study of voluntary environmental agreements found more substantive follow-through among firms involved in establishing the programme than among later joiners. The gap between joining dates was much larger than here. It supports giving preparation some weight; the size of the adjustment remains a judgment.
The central assumption is that OpenAI takes about 1.5 times as long as Anthropic. The uncertainty around that assumption includes a meaningful chance that OpenAI moves faster. Their probability of eventual implementation is the same. The chart also shows how the estimates change with easier or harder negotiations.

Calculation code · Sources and their use · Model assumptions and results
Liquidity over time
i Shows the market’s available liquidity over time. Liquidity can decay continuously and may change discretely if liquidity adjustments are applied.8 comments
Log in to comment.
OpenAI still has no operating embedded-evaluator engagement on the public record. On October 9, OpenAI's research leaders posted that the company is "actively finalizing contracts with third-party safety assessors and will announce details in the coming weeks." The same post says OpenAI is committed to embedding external assessors. It was a response to the letter from the three fired safety researchers. — OpenAI Newsroom on X, October 9
Reuters reported on October 13 that OpenAI and Anthropic had sought strict control over what evaluators could publish, as recently as October 5. OpenAI told Reuters that it does not seek approval over entire reports and lets evaluators disclose significant redactions. (summary of the Reuters report)
When METR published its summary on October 22, OpenAI described the July 21 – August 11 METR/Redwood exercise as a pilot "embedded evaluation". It said the findings will shape "future research collaborations with METR and other third party evaluators". No contract or start date was announced. (OpenAI on X, October 22)
Still no named OpenAI assessor as of October 28, nearly three weeks after the "coming weeks" statement. OpenAI's October 23 deal with Poland's NASK is a collaboration to test models, with no access or publication terms given (TNW). WSJ reported that the firings partly cited disappearing Signal chats with METR's Ryan Greenblatt, who denies leaking (TNW).
OpenAI told Transformer it is concerned about capacity for independent audits. METR’s Chris Painter says staff capacity, rather than access at frontier labs, constrains expansion; proposed evaluator training programs remain at least a year from enrollment. Transformer
OpenAI announced Phase 1 embedded assessments with Apollo Research, METR, Transluce and Goodfire (OpenAI). Apollo embeds six evaluators for four months with training/RL/checkpoint access and non-redaction of unfavorable findings (Apollo); Transluce starts a six-week onsite pilot in early December ahead of a broader early-2027 assessment (Transluce).
Transluce's own post says its access "is not comparable to 'employee-level' or 'employee-like' access." It cannot publish results on internal-only models, OpenAI decides further access requests, and OpenAI must approve its staff. Transluce
OpenAI told TechCrunch the pilots start at different times, with Apollo's starting this week. OpenAI pays Apollo and Goodfire; METR and Transluce are unpaid. OpenAI can still redact evidence it deems sensitive IP, but not Apollo's verdicts. The scope of METR's second assessment is still undecided. TechCrunch