One robots.txt rule to opt out of AI training: what it also blocks

On 30 October, Google said it will honour a shared rule that lets a website opt out of AI training with one short rule in robots.txt, the small text file that tells crawlers what they may read. Microsoft's Bing said the same on the same day, and OpenAI has committed to it. Until now, keeping your pages out of AI models meant naming every company's bot one by one. The new rule names none of them: one rule covers every company that adopts it. Before you ask whoever runs your site to add it, know three things. No company honours it yet. At Google, the same switch decides whether Gemini can cite your pages. And for a business whose pages exist to be found, opting out mostly protects text that was written to be repeated.

TL;DR: To opt out of AI training across providers, robots.txt will take User-Agent: __usage_ai-training followed by Disallow: /. Bing honours it from 1 February 2027, Google in early 2027. At Google it also removes your pages from Gemini app answers. Your Google Search ranking is untouched.

What is the new AI training opt-out in robots.txt?

A single entry, __usage_ai-training, written in robots.txt the way you would address a crawler, is what the new rule adds. The draft calls it a "policy token". It comes from an Internet Engineering Task Force draft published on 30 October 2026, titled Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00).

The draft defines AI training narrowly, as "using an asset to modify the learned parameters of a generative AI model". It states a preference and enforces nothing. The draft says plainly that the token is not a crawling directive: a company that honours it may still visit your pages, and it promises only not to train on them.

The difference from what exists today is the number of lines. Google has offered its own token, Google-Extended, since 2023. OpenAI, Anthropic and others each have their own bot names. A site that wanted out of training had to list every one of them and keep that list current as new bots appeared. The shared token is one entry that every participating company reads, and their company-specific tokens keep working alongside it.

The draft's own example is three lines that ask every participating company not to train on anything on the site except its press kit:

• User-Agent: __usage_ai-training
• Disallow: /
• Allow: /press-kit/

When do Google, Bing and OpenAI start honouring it?

Nobody honours the shared token today, so a line added this week changes nothing until 2027. Google's announcement says: "We aim to support this new control along with other AI providers in 2027", and its crawler documentation now says early 2027. Microsoft has given dates. Bing starts honouring the new preferences as it crawls from 1 February 2027. OpenAI has published a commitment to the signal without a start date.

The standard itself isn't finished either. The draft is version 00, and Nate Hake, who takes part in the IETF working group, said significant disagreements remain about parts of it. The token's name and syntax could still change before 2027.

If your developer adds the shared line this week, ask them to put a reminder in the calendar for February 2027 to check the syntax still matches what Google and Bing say they read. And if your robots.txt already blocks training bots by name (OpenAI's GPTBot, Anthropic's ClaudeBot, Common Crawl's CCBot, Google-Extended), keep those lines. The companies behind those names honour them today, and they keep working after the shared token arrives.

Does opting out of AI training remove you from Google Search or AI answers?

Opting out of AI training does not touch your Google Search ranking. Google's documentation says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal", and Microsoft says traditional Bing rankings are unaffected by its new settings. AI Overviews and AI Mode are part of Google Search, so the training token does not take you out of those either.

The exception is Gemini. Google-Extended, the switch the shared token will flip at Google, controls two things at once: whether Google may train future Gemini models on your pages, and whether the Gemini app may use your pages to ground its answers, which is where Gemini quotes and links its sources. Google's announcement spells out the consequence: if you use the shared token and don't mention Google-Extended, Google treats it as if you had blocked Google-Extended. And only "websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks via grounding in Gemini Apps".

At Google, then, there is no setting for "don't train on my pages, but do cite them in Gemini". You can opt out of training everywhere and allow Google-Extended separately, which keeps Gemini citations but lets Google train on your site. Or you can opt out everywhere and accept that the Gemini app won't link to you.

Bing has split the two jobs. Its new settings separate AI Training from AI Grounding, plus a Limited Grounding option that lets Microsoft's own products, such as Copilot, use your pages to answer questions while refusing it to the grounding service Bing offers other companies' AI products. On Bing, a business can refuse training and still be cited by Copilot.

Should a business website opt out of AI training?

Whether to opt out of AI training depends on what your pages are for. Ask whether your content is the thing you sell, or the thing that describes what you sell.

When the content is the product, the case for opting out is strong. A recipe site, a course creator, a photographer or a publisher earns from people reading the work itself, and a model that has learned it can answer without sending anyone to the source. For them, one rule covering every participating company is a real improvement, and the loss of Gemini citations may be a price worth paying.

For a business whose pages describe a service, the calculation changes. A dental practice, a guesthouse or an accountant publishes opening hours, areas served and what a first appointment involves so that people find them and book. Nobody pays to read those pages, so keeping them out of training protects text that was written to be repeated. At Google, it also gives up the Gemini app as a place where your live pages are quoted and linked, which matters more than training for a business whose details change: an answer grounded in your page can read today's version, while a trained model remembers whatever it saw.

Some service sites also carry work that is worth protecting: original photography, a paid-for guide, a detailed treatment explainer. The rule works per folder, like the draft's press-kit example, so you can keep those paths out of training and leave the rest of the site open.

That is the decision we made for our own site. sharphaw.com's robots.txt lets every crawler in, training bots included, and blocks only the technical /api/ routes. Every page on it exists to explain what we do to someone deciding whether to hire us. If we sold the writing itself, the answer would be different.

Two limits apply either way. An opt-out does not reach backwards: it applies to future training, not to models already built. And robots.txt only works on companies that choose to honour it. Anything that must stay private belongs behind a login.

What to do about it before 2027

For most owner-operated businesses there is nothing to change this month. The useful work now is deciding, so the change takes minutes once the token is live. In order:

1. Open your robots.txt. Type your domain followed by /robots.txt into a browser. You want to know what it says today, and who can change it: your developer, your website builder's settings, or nobody.
2. Decide what your pages are for. If the content is the product, plan to add the shared line. If the pages describe a service, the default of allowing training is defensible, and it is what we chose.
3. If you opt out, decide about Gemini separately. If being cited in the Gemini app matters to you more than staying out of Google's training, add an explicit allow for Google-Extended next to the shared line.
4. Keep any existing per-bot blocks. They work now, and the shared line does not replace them.
5. Check for Bing's noarchive or nocache tags. Most small sites don't have them. They are lines in a page's code that tell Bing not to keep a copy of the page, and some sites used them to stay out of Copilot. Microsoft is replacing them with separate training and grounding settings in Bing Webmaster Tools: temporary grounding controls appear there on 15 December 2026, and sites on the old tags have until 1 March 2027 to set the new preferences if they want to keep their restrictions.
6. Write down which option you chose and why. Whoever looks after your site in 2027 will need it when the token goes live.

The short version

The shared token turns the AI training opt-out into one rule, and it is a real gain for anyone whose writing is what they sell. Nothing reads it before 2027, and at Google it takes the Gemini app's citations with it. For a guesthouse, a clinic or a practice, staying open is a defensible default; what matters is that the owner chooses it on purpose.

Checking what a site tells crawlers, and changing it when rules like this go live, is part of the upkeep in our website plans; how your pages get found and quoted is the job of our Content Engine service. If you'd like us to look at yours, book a 30-min call.