Can you stop AI from training on your website? What the new shared opt-out means for small businesses

AI 2 Nov 2026 5 min read

Your website already talks to AI. AI companies send programs called crawlers to read public web pages. Some of what they read is used to train AI models. Some of it is used to answer a customer’s question in an AI assistant, often with a link back to you.

Until now, a website owner who didn’t want their pages used for AI training had to block each AI company’s crawler one by one. On 30 October 2026, that started to get simpler.

What was announced

On 30 October 2026, OpenAI, Google and Microsoft each said they will support one shared signal that tells AI companies, “don’t use my content for AI training.” It lives in robots.txt, the small text file on your website that tells crawlers what they may read.

One rule group for many AI companies. Instead of separate rules for every AI company, a site can add one group of rules. The proposal is an Internet-Draft submitted to the Internet Engineering Task Force (IETF), the body that writes internet standards. It is an individual draft, not yet an approved standard. It was written by Kevin Kelley of Anthropic.
What it looks like. The draft’s own example is:
```
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
```
This says “don’t train AI on my site,” except for the /press-kit/ folder.
It’s a preference, not a lock. The draft says this signal “is not a directive for crawling behavior.” It states what you prefer. It doesn’t physically stop a crawler from reading your pages. To limit crawling itself, you still use each crawler’s own rules.
It isn’t live yet. OpenAI says it aims to implement the signal “in the coming months.” Google says it aims to support it “in 2027.” Microsoft says it will honour its new preferences from 1 February 2027, as it crawls. Anthropic, where the draft’s author works, says it has started implementing support, and that adding it across its services will take time.
Most old rules keep working. OpenAI and Google both say their existing crawler controls continue alongside the new signal. Google says it plans to treat the shared signal as if you had blocked Google-Extended, its AI training control, unless your file mentions Google-Extended directly. Microsoft is the exception: after 1 March 2027, it will no longer read its older NoArchive and NoCache tags as training or grounding preferences.

Microsoft went one step further. It now separates AI Training (using your content to build future AI models) from AI Grounding (using your content to help write an AI answer, in places like Copilot and Bing). Both are allowed by default. Training can be set in robots.txt, but grounding is set page by page, not in robots.txt. Microsoft warns that if you block grounding, “you will see reduced brand visibility in AI experiences.”

Training vs answering: the part that matters for SMEs

These are two different things, and small businesses often mix them up.

Training is when an AI company uses your pages to teach a future model. The draft and the companies’ posts talk about preferences going forward. Anthropic, for example, says blocking its training crawler signals that your “future materials” should be left out. None of them says a block removes content already collected.

Answering (Microsoft calls it grounding) is when an AI assistant reads your page to answer a customer’s question right now, such as “Which kedai in Shah Alam delivers on Sunday?” This is how your business shows up in AI answers.

For a news site or a photographer, blocking training may make sense. For most Malaysian SMEs, your website exists so people can find you. If you block the wrong thing, AI assistants may stop mentioning you.

What to do this week

1. Look at your robots.txt. Type yourwebsite.com/robots.txt into a browser. See what’s already there. Many sites have rules a web designer added years ago that nobody remembers.

2. Decide on training and answering separately. With most AI companies, you can block training and still allow AI search. OpenAI lists separate crawlers for training (GPTBot) and for ChatGPT search (OAI-SearchBot). Anthropic lists separate crawlers for training (ClaudeBot) and for search (Claude-SearchBot). Google is different: its Google-Extended setting covers both training and answers in Gemini Apps, and Google says sites that allow Google-Extended can appear in responses and get clicks in Gemini Apps. Google also says Google-Extended doesn’t affect Google Search.

3. Don’t block AI search by accident. Anthropic says disabling Claude-SearchBot “may reduce your site’s visibility” in search results. Before you add a blanket block, ask your web person what each line does.

4. Make the pages AI reads worth reading. Clear prices, opening hours, delivery areas and policies help AI give the right answer about you.

Where ThinkChat fits

Getting found is step one. The next question is who answers when the customer messages you. ThinkChat is a multi-channel AI Chat Agent, not a chatbot. WhatsApp is the main channel in Malaysia, connected through Meta’s official WhatsApp Business Platform API or a QR scan, and ThinkChat also works on other channels like Instagram, Facebook Messenger and website chat. It answers from the information you approve, and hands the chat to a person when a customer needs one.

Soft next step

Want an AI Chat Agent to answer your customers 24/7? Talk to Lyia on WhatsApp: +60 11-5515 9317.

Sources

Checked 2 November 2026.

← All articles