If your author site hosts sample chapters, free short stories or a reader magnet, you may want a clear way to say that those pages should not be used to train AI models. A new robots.txt line is being proposed for that. It is a proposal, not a finished standard.
The lesson: decide which folders you want kept out of AI training, then paste one short group into robots.txt, and keep your older crawler rules in place.
What the proposal is
On 30 October 2026 an IETF Internet-Draft titled Machine-Readable Training Preference Signal was published (checked again on 12 November 2026). The datatracker banner says: "This I-D is not endorsed by the IETF and has no formal standing in the IETF standards process."
The draft defines a policy token named __usage_ai-training. Its own example is these three lines:
User-Agent: __usage_ai-trainingDisallow: /Allow: /press-kit/
The draft says this "expresses a preference against AI training for all paths except those under /press-kit/". A plain User-Agent: * group does not count. The token must be named.
The draft also says the token "is not a directive for crawling behavior" and that content-usage preferences and crawler rules "are independent and orthogonal". So this line is about training preference, not about whether Googlebot indexes your site for Search.
What Google says
Google's common crawlers page (checked again on 12 November 2026) says: "Google intends to support the AIPREF shared machine-readable training preference signal (with "user-agent: __usage_ai-training") in early 2027."
Google's Search Central blog post of 30 October 2026, Supporting a shared AI training control for websites, adds two points in Google's own words:
- "When a website uses this policy token to opt out of training in its robots.txt file (and Google-Extended is not explicitly specified), Google plans to interpret that as if the publisher had disallowed Google-Extended."
- "Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps."
The same blog says Google aims to support the control "along with other AI providers in 2027". Google-Extended itself "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
What this does not do
- It is a preference, not a lock. Sites that ignore robots.txt are not stopped by this line.
- It does not remove anything already collected or already used in a model.
- It does not replace your existing GPTBot, ClaudeBot or Google-Extended rules. Those keep working alongside it.
- Contract and settlement clauses are a different topic. Our 21 October post on AI-training clauses covers that side.
A five-minute check
- List the folders on your author site that hold sample chapters, free stories or reader magnets.
- List any folder you are happy to leave open, such as a press kit or blurbs page.
- If you want a site-wide preference against training, with one folder left open, paste the three-line example above and change
/press-kit/to your open folder. - Keep your older crawler rules. Do not delete Googlebot rules unless you mean to leave Search.
- If you care about appearing in Gemini app answers via grounding, read Google's two lines above again before you disallow Google-Extended, directly or through the new token.
- Write down your choice and the date next to your other AI records. Our one-page AI policy post is a place to file it.
If you are not sure where robots.txt lives on your host, ask the host or your site builder. Do not guess the path.
How this ties to Meta's terms
Our 10 November post on Meta's Commercial Terms covers Meta's pause of the 20 November Commercial Terms update. Meta's Business Help notice says its crawling practices remain unchanged and that content owners can continue to signal their crawling preferences using robots.txt. This post is about a separate proposed line for AI-training preference.
This week
Run steps 1 and 2 today. Once you know which folders matter, the paste itself takes a minute.