EVIDENCE FILE // OCTOBER 31, 2026

robots.txt AI Training Signal Evidence File: Three Documents and a Draft

Claim-level evidence for the report “OpenAI and Google Backed One robots.txt Opt-Out. The Draft Says It Stops No Crawl”. Every quotation in that article was extracted from the published page and required to appear verbatim in a saved copy of a primary document. This file records the captures, what each claim rests on, and the one thing the reporting cannot supply: a seat at the working group.

• Story OpenAI and Google Backed One robots.txt Opt-Out. The Draft Says It Stops No Crawl.
• Published October 31, 2026
• Documents captured October 31, 2026
• Check type Whole-article quotation extraction, plus named-quotation check with negative controls
• Result 24 substantive quotations checked, 24 found, 0 missing

Documents relied on

All three documents are self-published by the organisations making the claim, and all three were read in full. The draft is also the only one of the three that states its own status in the text, which is why it carries this story.

Two related drafts were checked for existence and returned HTTP 404: draft-ietf-aipref-attach-00 and draft-ietf-aipref-vocab-00. The article therefore describes the signal as one draft, not as a family of adopted documents.

Method

The primary check reads the published article rather than a hand-kept list. It extracts every span the page sets as a quotation, normalises markup, typographic punctuation, non-breaking spaces and zero-width characters, splits any span containing an ellipsis so each surviving fragment is tested separately, and then requires each fragment to appear verbatim in at least one captured document. Spans that quote one of the article’s own headings are declared in advance and listed, rather than silently ignored.

Result: 24 substantive quotations checked, 24 found, none missing, with one declared internal reference (“What we could not establish”). A second, named-quotation harness tested the same captures against a fixed list including seven negative controls — strings that must not be found, such as “OpenAI has implemented the signal” and “Google will honor the token in 2026”. All controls were correctly absent, which is what allows the pass to mean something.

Claim-level table

What this article does not claim

• It does not say the signal has shipped. OpenAI and Google both describe future implementation.
• It does not say the draft is an IETF standard, an IETF position, or adopted by the AI Preferences working group.
• It does not claim any AI company will comply beyond its own public words.
• It does not claim the token is unenforceable in law. It says the draft distinguishes preferences from crawl directives, and that private deals may override the mechanism.
• It does not name a Google author, because the post names none.

The gap this file cannot close

We read published documents. We were not in the working group, on the mailing list, or in any conversation among the participants, so nothing here should be read as reporting what the parties intend beyond what they published. The two 404s above are the limit of what a filename guess can establish about the other AIPREF drafts; other related drafts may exist under names we did not try.

Production record

← Back to the report