One robots.txt Line Now Opts Your Site Out of AI Training

If you run a blog, you’ve probably done this dance before. First you learned the name GPTBot and blocked it. Then Google-Extended. Then ClaudeBot. Then a few more, each with its own syntax, each announced in a different blog post, each quietly changing over time. Keeping your content out of AI training felt like maintaining a list of exes.

On October 30, 2026, that changed — or at least, the industry agreed it should. Google, Microsoft, OpenAI, and Anthropic all said they will honor one shared robots.txt entry that tells every participating company, in a single stroke, whether your pages may be used to train generative AI models. One line, four companies, no more per-crawler scavenger hunt.

Here’s the honest version up front: nothing is live yet. No company is reading the new token today. The rollout lands across 2027 on different dates, the standard itself is still a draft with no formal standing, and the token covers training only — not search, not AI answers, not crawling. If you add it now, you’re planting a flag for next year, not flipping a switch today. That’s still worth doing, and this article shows you exactly how, plus the three traps that catch most site owners.

The short answer: add two lines to robots.txt

The mechanism comes from an Internet-Draft titled “Machine-Readable Training Preference Signal,” written by Kevin Kelley of Anthropic and published on October 30, 2026. Its number is draft-kelley-aipref-training-preference-signal-00. The trick is elegant because it invents nothing new: it reuses the ordinary robots.txt group format, but the User-Agent line names a use rather than a crawler.

To opt your whole site out of AI training, add this group to your robots.txt file:

User-Agent: __usage_ai-training
Disallow: /

That is the entire mechanism. The token __usage_ai-training — two leading underscores included — is a policy token, not a bot. No crawler sends it as a user agent. Participating companies read the group and treat it as your stated preference about training. You can also carve out exceptions. The draft’s own example opts out of everything except a press kit:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

Google’s Search Advocate John Mueller put the point plainly in the Google Search Central announcement, “Supporting a shared AI training control for websites”: “Instead of managing separate instructions for every AI developer, website owners will be able to use a shared ‘policy token’ within robots.txt to set broad preferences across participating AI providers.”

That is the whole pitch. Now the fine print, because the fine print is where this story actually lives.

What each company actually promised (and when)

The four announcements landed the same day, but they are not the same promise. This is the part most coverage glosses over, and it matters for your planning.

Anthropic has already started implementing support, according to Kelley himself on the IETF’s AI Preferences working group mailing list — with the caveat that covering all of its services will take time. It is the only one of the four that can say “started” rather than “planned.”

Microsoft gave the firmest date. Its Bing Webmaster blog post says Bingbot will start honoring the new preferences as it crawls from February 1, 2027. Microsoft is also going further than the others: it is adding a separate, page-level grounding control — a meta tag or HTTP header that decides whether a page can feed the answers Bing, Copilot, and Microsoft 365 generate on the spot. Temporary transition settings for that control open in Bing Webmaster Tools on December 15, 2026, and sites relying on the older NoArchive or NoCache directives as AI signals have until March 1, 2027 to switch. The temporary grounding preferences expire September 30, 2027.

Google says it aims to support the control “along with other AI providers in 2027,” and its crawler documentation narrows that to early 2027. No exact date yet, and Google promises progress updates. One Google-specific detail matters a lot: if your site uses the shared token to opt out and does not explicitly mention Google-Extended, Google will treat that as if Google-Extended were disallowed. That covers Gemini training and, importantly, grounding in Gemini Apps and Vertex AI. Google also notes that sites which specifically allow Google-Extended remain eligible to appear in grounded responses and receive clicks from them. Opting out, in other words, can cost you that particular visibility.

OpenAI says it aims to implement the signal “in the coming months.” No date, no quarter. Meta has voiced support for the idea but has not committed to honoring the token. If Meta’s crawlers matter to your opt-out strategy, the old per-company rules are still your only lever there.

Here is the timeline in one place:

The three traps: what the token does NOT do

Trap 1: Your existing “block everything” robots.txt says nothing

This one is genuinely counterintuitive, and it is the trap most likely to bite experienced site owners. The draft deliberately excludes wildcard groups: a User-agent: * group must not be interpreted as matching the policy token. So if your robots.txt currently shuts out every bot with a blanket disallow, that file expresses no AI-training preference under the new system. You have to add the __usage_ai-training group by name, even if you already block everything else. Think of it as a “no soliciting” sign that the new rules explicitly say does not cover one particular salesman. If you want out, name the token.

Trap 2: It covers training, not AI answers

The token states a preference about one specific use, defined in the draft as “using an asset to modify the learned parameters of a generative AI model.” That is training. It does not block crawling. It does not remove your pages from search results — Google is explicit that Google-Extended “does not impact a site’s inclusion in Google Search,” and the shared token inherits that treatment. And it does not stop an AI system from quoting your live pages in an answer. That last job belongs to separate controls: Google’s AI search opt-out options and the AI Overviews opt-out for publishers, plus Microsoft’s new grounding preference, which is exactly why Microsoft built a second control alongside the training one. If your goal is “don’t let AI answers summarize my work,” the training token alone will not get you there. We covered Google’s AI search opt-out options separately, and they remain a different decision from this one.

Trap 3: It is voluntary, and it is not retroactive

The draft is an individual submission. It has no IETF endorsement and no formal standing in the standards process, and it expires on May 3, 2027 unless it is revised or adopted. The companies’ commitments are voluntary statements, not law — though Google notes the proposal fits the EU’s Code of Practice for general-purpose AI models under the AI Act, which treats robots.txt as a primary way for publishers to state machine-readable preferences. Also, the draft judges preferences against the robots.txt that was in effect when a page was fetched. It does not reach back to copies already collected. Adding the token today does not un-train anything.

None of this makes the token useless. It makes it a statement of intent for future training runs, adopted by the four companies whose models matter most, in a format that future companies can adopt without inventing their own token. But go in with your eyes open: this is a preference signal, not a lock on the door.

Should you opt out, opt in, or split the difference?

There is no universal right answer, and anyone selling you one is skipping the trade-offs. Here is how I would think about it for a typical independent blog or small business site.

Opt out of everything if your content is the product itself: original reporting, a paid course’s free samples, a book’s chapters, photography or illustrations, proprietary data or research. If a model trained on your work can answer the exact questions people currently visit you to answer, training is a substitute for your site, not a discovery channel for it.

Stay in (or say nothing) if your business runs on discovery and the AI-answer visibility is worth more than the training risk: local service pages, affiliate roundups, how-to content where the answer is widely known anyway. Note that saying nothing is itself a choice — a robots.txt with no __usage_ai-training group expresses no preference at all.

Split it with Allow rules if your site has both kinds of content. This is the option most site owners overlook, and it is the one the draft’s press-kit example is really demonstrating. A news site might disallow training on its investigations while allowing it on press releases. A course seller might disallow training on lesson content while allowing it on the sales pages it wants quoted everywhere. Path-level control is the whole point of reusing robots.txt grammar: longest-match precedence and ordinary Allow/Disallow rules already work, and parsers that do not recognize the token simply ignore the group.

One more consideration, and it is a business one rather than a technical one: the Google grounding trade-off. Because Google reads a shared-token opt-out as a Google-Extended disallow (unless you mention Google-Extended separately), opting out can remove your pages from grounded responses in Gemini Apps and Vertex AI — the responses that carry links and clicks. If those clicks matter to you, you can keep Google-Extended explicitly allowed while opting out of the shared token for everyone else. That is a legitimate, if fiddly, combination, and it is worth deciding deliberately rather than stumbling into it.

How to add it on WordPress (and check your work)

For most WordPress sites, the fastest route is your SEO plugin. In Rank Math, go to Rank Math SEO → General Settings → Edit robots.txt (the exact menu label varies by version) and add the group at the end of the file. If you prefer the file itself, edit the robots.txt in your site root over FTP or your host’s file manager. Either way, the group looks like this for a full opt-out:

User-Agent: __usage_ai-training
Disallow: /

For a partial opt-out — say, keeping your public press page available — add the exception below the disallow:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press/

Then verify. Visit yoursite.com/robots.txt in a browser and confirm the group is there, spelled exactly — two underscores, usage_ai-training, no typos, no trailing spaces that matter. Then do the step people skip: keep your existing per-company rules in place for now. The shared token is not honored by anyone yet, and Meta has not committed at all, so your GPTBot, ClaudeBot, and Google-Extended groups are still doing the actual work until the 2027 rollout lands. Removing them early would leave you with less protection, not more. Think of the new group as the future-proof layer, not a replacement.

Set yourself a calendar reminder for February 2027. That is when Bing starts honoring the token and when Google’s early-2027 window opens — the moment to re-check that the token is still in your file, that Google’s documentation has firmed up its date, and that no plugin update has overwritten your robots.txt. Robots.txt files get clobbered by plugin updates more often than anyone admits.

Test it before you trust it

One practical habit separates site owners who actually benefit from new robots.txt controls from those who just collect them: testing. After you add the group, check it three ways. First, load yoursite.com/robots.txt and read the group with your own eyes — plugins and caching layers both have a habit of serving a stale file. Second, run it through a robots.txt tester: Google Search Console’s tester validates syntax against Googlebot’s parser, and Bing Webmaster Tools has its own equivalent. A single missing underscore in __usage_ai-training turns the whole group into a comment that no company will read, and testers catch that instantly. Third, check that your caching plugin or host has not pinned an old robots.txt in cache — purge the cache after editing, then re-fetch the file in a private browser window.

There is a second, subtler test worth running: fetch your robots.txt as each of the old crawlers would see it. Your file now carries two layers — the old per-company groups and the new shared group — and they can contradict each other. If your GPTBot group says Disallow and your shared group says Allow for the same path, the company-specific rule is the more specific one and should win for that company, but the shared group governs everyone else. Contradictions are legal in robots.txt; they are just confusing, and confusion is how a press page ends up blocked or a paywalled lesson ends up allowed. Read the finished file top to bottom once, as a stranger would, before you call it done.

The old stack, and when to retire it

Until the rollout completes, your robots.txt probably carries a stack like this: GPTBot, ChatGPT-User, Google-Extended, ClaudeBot, anthropic-ai, CCBot, and a few others, each learned from a different announcement at a different time. The shared token is designed to replace that stack eventually — one group instead of a dozen, with newcomers inheriting the convention as long as they adopt it. The honest timeline for retiring the old rules is “when the companies you care about have all confirmed they honor the token, and you have verified it.” For most sites, that means late 2027 at the earliest, and Meta’s absence means at least one major crawler still needs its own rule indefinitely. Do not delete the old groups in a fit of tidiness this week.

It is also worth noting what this announcement is not. It is not a settlement of the AI training copyright lawsuits, it is not a licensing deal, and it does not pay anyone. Publishers who want compensation for training use still need licenses and partnerships; this token is a free, voluntary preference signal. The same week, Reddit announced it would move toward a mostly logged-in site partly to protect the posts it licenses to AI developers — the opposite strategy, a wall instead of a signal. Both approaches will coexist, and which one fits your site depends on whether you have something to license or something to protect. (For the other side of the training debate, see how Meta softened its AI training terms in October.) For the full announcement details, see the shared-token explainer, the four-company timeline, and Google’s official announcement.

What to do this week

If you take one action from this article, make it this: add the __usage_ai-training group to your robots.txt now, keep your existing per-company rules, and set a February 2027 reminder to verify the rollout. The cost is two lines of text. The benefit is that when Google, Microsoft, and OpenAI switch their support on, your preference is already on file, in the format they agreed to read, instead of something you scramble to add after the fact.

The bigger story here is easy to miss in the syntax. For three years, every AI company invented its own opt-out, and site owners paid the complexity tax. On October 30, the four biggest players agreed to read the same line. That is a small, boring, genuinely useful piece of internet infrastructure — the kind that only matters once everyone uses it. Be early. Two lines, five minutes, done.

Frequently Asked Questions

Does adding the AI training opt-out hurt my Google rankings?

No. Google states that Google-Extended does not impact a site’s inclusion in Google Search, and the shared token is treated the same way for search. What can change is grounding visibility: a shared-token opt-out is read as a Google-Extended disallow, which can remove your pages from grounded responses in Gemini Apps and Vertex AI unless you explicitly allow Google-Extended.

When will Google, Microsoft, and OpenAI actually start honoring the token?

Microsoft starts on February 1, 2027. Google aims for early 2027 and has given no exact date. OpenAI says “in the coming months.” Anthropic says it has already started implementing support across its services. Nobody honors it yet as of October 31, 2026.

I already block all bots with User-agent: *. Do I still need the new group?

Yes. The draft deliberately excludes wildcard groups from matching the policy token, so a blanket block expresses no AI-training preference. You must add a group naming __usage_ai-training explicitly.

Does the token stop AI chatbots from quoting my pages in answers?

No. It covers training — using content to modify a model’s learned parameters. Live answers that quote your pages are governed by separate controls, such as Google’s AI search opt-outs and Microsoft’s page-level grounding preference. If you want to block answer usage, you need those controls too.

Should I delete my GPTBot and Google-Extended rules now?

No. Keep them until every company you care about confirms it honors the shared token, and Meta has not committed to it at all. The new group is an additional, future-proof layer — not a replacement yet.

What happens if the IETF draft expires in May 2027?

Internet-Drafts expire after six months unless revised. The May 3, 2027 expiry means the working group must update or adopt the draft by then; it does not automatically cancel the companies’ commitments, but it is a date worth watching. If the draft lapses without adoption, the token’s future depends on the companies continuing to honor it voluntarily.