An Outside Red Team Found Ways Around OpenAI's Own Agent Monitors
Article: reports/fdw-run-001-story-d-metr-article.md Evidence checked: October 27, 2026 Primary source: METR, "Red-Teaming OpenAI's Internal Agent Monitoring Systems," October 22, 2026 URL: https://metr.org/blog/2026-10-22-red-teaming-openai-agent-monitoring/ Retrieval: HTTP 200, 70,967 raw bytes, 12,106 characters extracted text Local copy: tmp/metr_openai_agent_monitoring.txt Verification script: tmp/verify_story_d.ps1 — 28 assertions passed, 0 failed, exit 0 Evidence grade: Primary source retrieved in full from origin and read directly by the controller. All quotations machine-verified against the retrieved text.
Verification method
tmp/verify_story_d.ps1 re-reads the saved METR text and the article, normalises curly quotes and dashes to ASCII equivalents and strips markdown emphasis, then asserts:
- 11 verbatim quotations present in both the source and the article;
- 7 figures present in both;
- 2 canary guards (see §Canary below);
- 4 negative assertions — the article must not contain phrases making claims we cannot support;
- 4 positive assertions — required disclosures must be present;
- 2 encoding guards confirming both files decoded cleanly.
28 passed, 0 failed. Exit 0.
A defect in the first version of this verifier — recorded, because it produced false results
The first run reported 5 failures. Four were false. The verifier used Get-Content -Raw, which decodes using the system codepage rather than UTF-8; on a UTF-8 file written without a BOM this produces mojibake, so a needle like 75-94% could not match 75–94%. I initially misread this as file corruption and reported it as such. Reading the raw bytes with [IO.File]::ReadAllText($path, [Text.Encoding]::UTF8) showed zero mojibake in every report file — the corruption was in the diagnostic, not the artifacts. The verifier now reads UTF-8 explicitly and carries encoding guards that fail loudly instead of reporting phantom content mismatches.
One failure in that first run was real and is fixed: the article omitted the 38-page OpenAI-redacted version.
Four prose edits were made to satisfy verbatim fidelity: "fewer than" → "less than" for the 0.2% error rate; the compaction sentence realigned to the source's own wording; the 61-page report description made explicit; the 38-page redacted version added.
Claim-level verification
| # | Claim in article | Source statement | Status |
|---|---|---|---|
| 1 | Exercise ran July 21 – August 11, three weeks | "For three weeks, from July 21st to August 11th, a METR employee and a Redwood Research employee conducted a pilot high-access exercise" | CONFIRMED verbatim |
| 2 | Two researchers, one METR and one Redwood | same sentence | CONFIRMED |
| 3 | Access to internal codebase, monitor prompts/config, internal documentation | "had access to OpenAI's internal codebase, misalignment-monitor prompts / configuration details, and internal documentation" | CONFIRMED |
| 4 | Roughly a dozen interviews across monitoring, safety-systems, security | "conducted technical interviews with approximately a dozen researchers and engineers across the monitoring, safety-systems, and security teams" | CONFIRMED |
| 5 | OpenAI-issued devices, never left premises, accompanied onsite | "they worked on OpenAI-issued devices, which never left the premises, and were accompanied while onsite" | CONFIRMED (note: source uses "they", not "we" — the article paraphrases without quotation marks) |
| 6 | Not for robust public transparency | "not to provide robust public transparency about OpenAI's monitoring systems" | CONFIRMED verbatim |
| 7 | 61-page internal report | "The output of this exercise was a 61-page internal report" | CONFIRMED |
| 8 | 38-page version redacted by OpenAI, shared with leadership | "A 38-page version redacted by OpenAI was shared with the leadership teams at METR and Redwood." | CONFIRMED (footnote) |
| 9 | System as it stood on and before July 21, 2026 | "the internal misalignment monitoring system as it existed on and before July 21st, and did not evaluate changes that were in progress at the time of the exercise" | CONFIRMED |
| 10 | OpenAI expanding during exercise; findings describe historical system | "describe the historical system we assessed on and before July 21st, rather than the expanded system's current coverage, which this exercise did not establish" | CONFIRMED verbatim |
| 11 | 75–94% of agentic internal Responses API traffic eligible, models ≥ GPT-5.5 | "roughly 75–94% of agentic internal Responses API … traffic from models at least as strong as GPT-5.5 was eligible for monitoring" | CONFIRMED |
| 12 | Substantial uncertainty in both directions | "although we have substantial uncertainty in both directions" | CONFIRMED verbatim |
| 13 | 99.2% of internal Codex-agent traffic eligible | "out of the subset of Responses API traffic used for internal Codex agents, 99.2% was eligible for monitoring" | CONFIRMED |
| 14 | Eligible ≠ monitored; load shedding, errors, bugs may intervene | "is intended to be sent to a monitor, but in principle load shedding, monitor errors, or implementation bugs might prevent some eligible traffic from making it through the entire monitoring pipeline" | CONFIRMED verbatim |
| 15 | Compaction events unmonitored; ~10% of eligible Codex traffic | "not monitoring compaction events likely led to on the order of 10% of eligible internal Codex traffic not being monitored between May and July 2026" | CONFIRMED |
| 16 | OpenAI staff found the compaction gap independently and fixed it during the exercise | "OpenAI staff had independently identified this before the exercise began, and fixed this during the exercise" | CONFIRMED |
| 17 | Monitor triggered at turn end; agents could postpone monitoring indefinitely | "the monitoring system triggered when agents returned control to the user, so during the time period when compaction events were not being monitored, agents could indefinitely postpone being monitored by simply continuing to take actions" | CONFIRMED verbatim |
| 18 | Bypass Codex sandboxing and Auto-Review; any action the host user could take | "We found a way Codex agents could in principle bypass Codex sandboxing and Auto-Review, allowing them to take any action that the host user could take." | CONFIRMED verbatim |
| 19 | Exploit prevents hundreds of recent actions from being monitored; details redacted | "an exploit agents could use that would prevent hundreds of recent actions (including the actions involved in the exploit itself) from being monitored [exploit details redacted by OpenAI]" | CONFIRMED |
| 20 | Serialization differences between agent view and monitor view | "exploiting serialization differences between the trajectory agents see and what the monitors see" | CONFIRMED |
| 21 | Two client-controllable headers disable monitoring for new agents; act could be caught | "two independent client-controllable headers that disable the monitoring system for new agents … the act of setting these headers and creating a new agent instance could be caught" | CONFIRMED |
| 22 | Two unmonitored "read-only" tools; exploit not validated in practice | "We did not validate this behavior or test whether it was exploitable in practice." | CONFIRMED verbatim |
| 23 | METR could not check whether exploit was used in practice | "Due to access constraints, we did not have the ability to check whether agents had, in practice, exploited this vulnerability" | CONFIRMED verbatim |
| 24 | Monitor shown full trajectory, few minor exceptions | "with a few minor exceptions, the monitoring system is shown the full trajectory that agents see" | CONFIRMED verbatim |
| 25 | Prompts look for scheming, subversion, sabotage, exfiltration, deception, injection | "scheming, monitor system subversion, sabotage, data exfiltration, deception, and inbound/outbound prompt injection" | CONFIRMED |
| 26 | Errored on less than 0.2% of requests | "the monitor errored on less than 0.2% of requests" | CONFIRMED |
| 27 | Zero load-sheds in one 24-hour period | "in one 24-hour period during the exercise there were zero load-sheds" | CONFIRMED |
| 28 | OpenAI states production-impacting vulnerabilities patched | "OpenAI has stated that all of the vulnerabilities we identified that impact their production safeguards stack have been patched" | CONFIRMED verbatim |
| 29 | METR has not evaluated those claims | "We have not evaluated these claims." | CONFIRMED verbatim |
| 30 | OpenAI had sole-discretion redaction right | "had the right to redact from this public version any information that OpenAI (in its sole discretion) determined to be confidential or otherwise sensitive" | CONFIRMED verbatim |
| 31 | METR had no unilateral right to describe redactions | "did not have a unilateral right to describe such redactions" | CONFIRMED verbatim |
| 32 | METR endorses each finding presented | "we endorse each of the findings that are presented in this write-up" | CONFIRMED |
| 33 | Coverage figures exclude internal research inference service | "does not include other sources of internal agentic coding traffic … such as agents running on OpenAI's internal research inference service—we were unable to estimate the volume" | CONFIRMED (footnote) |
| 34 | RL training runs not covered at start; subset monitored March–June | per report body and footnote | CONFIRMED |
| 35 | Design principles as summarised | per the report's closing section | CONFIRMED — summarised, not quoted; no quotation marks used |
| 36 | METR stress-tested Anthropic's monitoring March 2026 | METR index lists 2026-03-25-red-teaming-anthropic-agent-monitoring; report's own footnote references "our previous stress-testing exercise with Anthropic" | CONFIRMED — date from METR's own URL slug and index entry; footnote confirms the prior exercise |
| 37 | Anthropic published auto-mode red-teaming research the same day | https://alignment.anthropic.com/2026/red-teaming-auto-mode/, October 22, 2026 | CONFIRMED |
Negative assertions — what the article must not say
Machine-enforced in the verifier, all passing:
- Must not frame METR's work as a response to Anthropic. The article states they are "contemporaneous rather than sequential."
- Must not claim OpenAI's fixes were verified. The article closes: "Neither METR nor this newsroom has verified that."
- Must not present the coverage figures as current. The article states the findings describe the system "as it stood on and before July 21, 2026" and quotes METR on not establishing current coverage.
- Must not use the canary or training-prohibition strings present in the source page.
Canary guard
The METR page carries a data-canary identifier and a training-prohibition notice. Both are deliberately not reproduced in the article, and the verifier asserts their absence. Content from the page is quoted for reporting only.
Outstanding uncertainties
- OpenAI's remediation is unverified. METR did not evaluate it; we did not either. The article says so twice.
- The public account is a redacted subset. OpenAI held sole discretion over redactions and METR could not describe what was removed. The 61-page report and the 38-page redacted version are not public. We cannot know what was withheld.
- Percentages are estimates with wide error bars, stated by METR as "substantial uncertainty in both directions."
- The compaction gap estimate (~10%) is partly reconstructed: METR notes the information needed to compute it precisely had not historically been logged.
- Five findings were not all validated in practice. METR explicitly did not test whether the "read-only" tools exploit was exploitable.
- No independent replication. One external team, one exercise, one lab's cooperation. No second red team has published a comparable assessment of the same system.
- No one from METR, Redwood or OpenAI was interviewed. Every quote is from the published post.
- "Open problem" framing belongs to Anthropic's paper, not METR's. Deliberately not imported into this article.
- Prior exercise date (March 25, 2026) is taken from METR's URL slug and index listing, not from a re-read of that page's own body.
Disputed or contested
Nothing in the article is contested by a second source, because no second source on this specific exercise exists. That is a limitation, not a strength: the article rests on a single institutional account, corroborated only by the fact that OpenAI's right of review is disclosed by METR and that METR states it endorses every finding published.
Corrections to earlier work in this run
- The Story D research update initially concluded no publication candidate should be created, on a chronological test that only counted documents published after 22 October. That conclusion was withdrawn once METR's same-day report was retrieved. Recorded in
reports/fdw-story-d-research-update-2026-10-27.md, Revisions 2 and 3. - A delegated worker asserted a Screwtape funnel containing a "17" step. No such step exists in arXiv:2610.20941. Recorded in ledger §22.