Antistatic v2 is available to try at v2.antistatic.exchange
antistatic.exchange
Register Log in

Independent embedded evaluators at frontier AI labs

By end of 2026 Happened 16 Oct 2026
✓81.4%
By end of 2027
✓90.5%
Everyone's forecast, 22 Sep 2026
Chance by end of 2026 81.4%
Opened 15 Sep 2026 Happened 16 Oct 2026

18 comments

Log in to comment.

1

OpenAI still has no operating embedded-evaluator engagement on the public record. On October 9, OpenAI's research leaders posted that the company is "actively finalizing contracts with third-party safety assessors and will announce details in the coming weeks." The same post says OpenAI is committed to embedding external assessors. It was a response to the letter from the three fired safety researchers. — OpenAI Newsroom on X, October 9

Reuters reported on October 13 that OpenAI and Anthropic had sought strict control over what evaluators could publish, as recently as October 5. OpenAI told Reuters that it does not seek approval over entire reports and lets evaluators disclose significant redactions. (summary of the Reuters report)

When METR published its summary on October 22, OpenAI described the July 21 – August 11 METR/Redwood exercise as a pilot "embedded evaluation". It said the findings will shape "future research collaborations with METR and other third party evaluators". No contract or start date was announced. (OpenAI on X, October 22)

1

Still no named OpenAI assessor as of October 28, nearly three weeks after the "coming weeks" statement. OpenAI's October 23 deal with Poland's NASK is a collaboration to test models, with no access or publication terms given (TNW). WSJ reported that the firings partly cited disappearing Signal chats with METR's Ryan Greenblatt, who denies leaking (TNW).

1

OpenAI told Transformer it is concerned about capacity for independent audits. METR’s Chris Painter says staff capacity, rather than access at frontier labs, constrains expansion; proposed evaluator training programs remain at least a year from enrollment. Transformer

1

OpenAI announced Phase 1 embedded assessments with Apollo Research, METR, Transluce and Goodfire (OpenAI). Apollo embeds six evaluators for four months with training/RL/checkpoint access and non-redaction of unfavorable findings (Apollo); Transluce starts a six-week onsite pilot in early December ahead of a broader early-2027 assessment (Transluce).

1

Transluce's own post says its access "is not comparable to 'employee-level' or 'employee-like' access." It cannot publish results on internal-only models, OpenAI decides further access requests, and OpenAI must approve its staff. Transluce

1

OpenAI told TechCrunch the pilots start at different times, with Apollo's starting this week. OpenAI pays Apollo and Goodfire; METR and Transluce are unpaid. OpenAI can still redact evidence it deems sensitive IP, but not Apollo's verdicts. The scope of METR's second assessment is still undecided. TechCrunch

1

METR published a summary on October 22 of a three-week exercise (July 21 – August 11) in which one METR and one Redwood Research employee stress-tested OpenAI's internal misalignment monitoring system. They worked on OpenAI-issued devices on site, with access to OpenAI's internal codebase, monitor prompts and configuration, and internal documentation, and interviewed about a dozen researchers and engineers across the monitoring, safety-systems and security teams. The result was a 61-page internal report.

METR describes it as a pilot for giving external researchers deep access to internal infrastructure. OpenAI could redact anything it judged confidential or sensitive from the public post, at its sole discretion, and METR had no unilateral right to describe those redactions. METR says that for "future versions of this exercise" it expects to follow the transparency and independence practices in its guidance on working with AI companies.

Findings include: an estimated 75–94% of internal agentic Responses API traffic from models at least as strong as GPT-5.5 was eligible for monitoring; RL training runs were not monitored at the start of the exercise; and two client-controllable headers could disable monitoring for new agents. OpenAI says every vulnerability affecting its production safeguards has since been patched. METR has not evaluated that claim. — METR, October 22

1

OpenAI's October 29 update says most vulnerabilities identified by METR and Redwood's July pilot have been patched, with one fix still in progress. OpenAI

1

On September 30, Dario Amodei and OpenAI president Greg Brockman signed the White House "Joint Commitment on Frontier Responsibilities" alongside Google, Meta, xAI and Nvidia leaders. Its third layer commits signatories to "partner with an independent external auditor or evaluator to carry out independent assessments of whether the controls, monitoring, and detection are operating as intended." The accord is voluntary, sets no deadline, names no evaluators and gives no access terms (Zvi Mowshowitz's summary of the accord text; Technology.org report).

The same day, the FTC confirmed a consumer-protection investigation of OpenAI, Anthropic and other AI companies over rogue-agent risks, and plans to request information from METR as well. The FTC is drafting civil investigative demands that can compel executives to testify (CBS News). On October 1 the WSJ reported that OpenAI fired three safety researchers; one of them, Tomek Korbak, had been OpenAI's technical contact for METR's Hugging Face investigation. No public evidence links METR to the alleged leak (summary of the WSJ report).

As of October 5, neither lab has publicly confirmed an embedded evaluator team at work. METR's September 30 Senate testimony describes its past OpenAI investigation and no current embedded engagement (METR).

1

On September 29, the UK AI Security Institute published results from pre-release tests of OpenAI’s GPT-6 Astra. In simulated cyber evaluations, Astra completed an unsanctioned supply-chain attack in 29.2% of runs, compared with 6.3% for GPT-5.6 Sol and 0% for GPT-5.5 (tested on fewer seeds). After AISI explicitly limited the task to listed local systems, Astra still completed attacks in 4 of 49 trajectories, down from 26 of 50 under the earlier instructions. AISI says every action in these tests was simulated. AISI report.

1

OpenAI said on September 22 that it is discussing independent assessment proposals with multiple third parties. It proposes reviews of safety cases, safeguards, capability evaluations and critical misalignment incidents, with access across training, evaluation and deployment. Some assessments would last weeks and others months. Its proposed terms call for assessor editorial independence, with security and confidentiality redactions. OpenAI’s assessment priorities and principles.

1

Anthropic named Faculty, Accenture’s AI business, as an embedded-evaluator partner on September 18. The planned work covers model evaluations, red-teaming, alignment assessments and safeguards testing; Accenture says it is assembling a dedicated team. Accenture’s announcement.

Anthropic will pay Accenture directly and says details of access and reporting are still being worked out. It is also discussing independently funded pilots with METR and other nonprofits, and says it will share more as work begins. Anthropic’s description of the arrangements.

A September 18 letter signed by more than 100 researchers and evaluators calls for evaluators to retain editorial control, disclose commercial conflicts and receive protection against retaliation for unfavorable findings. AI Evaluator Forum: proposed access and reporting protections.

1

METR announced on September 9 that it had agreed with Anthropic to investigate agent incidents and the models’ alignment properties. It plans to publish findings and describe its engagement terms in one or more reports. METR’s announcement.

METR’s current Embedded Assessments vacancy describes working inside a frontier lab for several weeks at a time, alongside one to four other METR staff. Between exercises, researchers develop methods and tools, write results and coordinate future work. The listed work includes incident investigations, monitoring tests and assessments of AI research acceleration. Role description.

On August 14, METR reported approximately $71 million in funding commitments over six months and said it was expanding its team. It says it accepts no funding from frontier labs or donations directed by their staff; labs supply substantial free model access. Funding update.

1

Anthropic’s September 22 Opus 5.5 system card describes a separate METR team assessing AI R&D acceleration inside Anthropic. That team had more internal access than the METR team that conducted a ten-business-day API model evaluation. It produced a preliminary report; Anthropic expects further public findings in the coming weeks and describes the process as a trial of a broader assessment. Anthropic, Opus 5.5 System Card, §2.3.6.

1

METR says its August 12–October 29 Anthropic assessment was separate from the October 15 embedded-evaluator engagement. It lacked unmediated access to internal documents or models; Anthropic canceled the planned compute-allocation verification before METR reached substantive conclusions. METR