Quick answer
You can control much of how your content is used in search snippets and AI answers, but each method covers a different scope. Snippets in Google Search and AI Overviews and AI Mode are controlled with nosnippet, data-nosnippet, max-snippet, and noindex, and collection for AI model training is controlled with per-bot robots.txt rules for bots such as GPTBot and ClaudeBot. However, following robots.txt is voluntary for crawlers, and blocking can also reduce your visibility.
Key takeaways
- nosnippet blocks all snippets for the page.
- data-nosnippet excludes only the specified text from snippets.
- Google-Extended does not affect inclusion or ranking in Google Search.
- GPTBot is the training bot, and OAI-SearchBot is the ChatGPT search bot.
- Following robots.txt rules is voluntary for crawlers.
How do you block Google Search snippets?
Google Search snippets are controlled with three settings: nosnippet, data-nosnippet, and max-snippet. nosnippet blocks all snippets for the page, data-nosnippet excludes only the text you specify, and max-snippet limits the length of the snippet.
To keep only certain paragraphs, not the whole page, from being excerpted, the right method is to add the data-nosnippet attribute to those parts. If you want to reduce only featured snippets, the guidance is to set max-snippet to a short length.
To remove the page itself from search results, use noindex. Unlike snippet controls, noindex removes search visibility altogether, so use each according to its purpose.
| Method | Applies to | Effect |
|---|---|---|
| nosnippet | Google Search (including AI features) | Blocks all snippets for the page |
| data-nosnippet | Google Search (including AI features) | Excludes only the specified text from snippets |
| max-snippet | Google Search (including AI features) | Limits snippet length; a short value reduces featured snippets |
| noindex | Google Search (including AI features) | Removes the page from search results |
| Blocking Google-Extended | Gemini training and grounding | No effect on inclusion or ranking in Google Search |
| Blocking GPTBot | Collection for OpenAI model training | Signals that the content should not be used for training |
| Blocking OAI-SearchBot | ChatGPT search | Not shown in search answers (may still appear as a navigational link) |
| Blocking ClaudeBot | Collection for Anthropic model training | Signals exclusion from future training data |
Can I block use only in Google AI Overviews and AI Mode?
AI features in Google Search are controlled with the same methods as regular search. Google says control over AI Overviews and AI Mode is done through robots.txt rules for Googlebot and through nosnippet, data-nosnippet, max-snippet, and noindex.
Google-Extended is often confused with these. Google-Extended is a control token for Gemini model training and grounding, and it does not affect inclusion or ranking in Google Search. It has no separate HTTP user agent and is used only as a robots.txt token.
In other words, blocking Google-Extended is not a way to control AI Overviews in Google Search. Also keep in mind that if you use snippet rules to reduce use in AI features, snippets in regular search results may be affected too.
How does this apply to ChatGPT and Claude?
Set separate robots.txt rules for each bot name the companies have published. Training bots and search bots are separate, so you can set things up so your site appears in search but is not used for training.
For OpenAI, GPTBot is the bot that collects content that may be used for model training, and OAI-SearchBot is the bot for showing sites in ChatGPT search results. The two settings are independent, so you can put 'User-agent: GPTBot' and 'Disallow: /' in robots.txt and still allow OAI-SearchBot.
Anthropic distinguishes ClaudeBot for training, Claude-SearchBot for search, and Claude-User for user requests, and says its bots follow robots.txt. Bot names and behavior are as of writing (2026) and may change, so check each company's official documentation.
Does blocking with robots.txt block everything?
No, it cannot be considered a complete block. Following robots.txt rules is voluntary for crawlers, and the rules may not apply to agents that fetch pages at a user's request.
OpenAI explains that robots.txt rules may not apply to ChatGPT-User because its actions are initiated by users, and Perplexity says Perplexity-User generally ignores robots.txt rules.
Google also explains that robots.txt is not a way to hide a page from Google. If the content must not be public, the right approach is to use noindex or protect it with a password.
What should you weigh before blocking?
Blocking is a decision that trades away visibility, so first identify whether what you want to stop is snippet excerpts, citation in AI answers, or model training.
Answer engine optimization (AEO) and generative engine optimization (GEO) aim to get content used in answers, which is the opposite direction from blocking. Visits earned through search engine optimization (SEO) may also be affected, so the common approach is to apply restrictions only to paid content or certain sensitive areas and leave the rest open.
Frequently asked questions
What is the difference between noindex and blocking with robots.txt?
robots.txt is a file that tells crawlers which URLs they can access, while noindex is a rule that removes a page from search results. Google explains that robots.txt is mainly for avoiding overloading a site with requests, so to keep a page out of search, use noindex or password protection.
When do changes to these settings take effect?
It differs by service. Google says changes can take from a few hours to several months, and the rules take effect after the page is crawled again. For AI services, it is best to check each company's official documentation for how long changes take.
Can content that has already been used for training be taken back?
Blocking with robots.txt is a statement of intent about future collection. Anthropic describes blocking ClaudeBot as a signal to exclude content from future training data. How already collected content is handled may differ by company, so check each company's policy.
Key entities in this article
- Google ( Organization )
- The company behind Google Search, AI Overviews and AI Mode, which publishes official SEO guidance through Search Central. Official source: Google
- AI Overviews ( Concept or service )
- The Google Search feature in which AI summarizes multiple sources and shows them with links. Official source: AI Overviews
- AI Mode ( Concept or service )
- The conversational AI search experience in Google Search for follow-up questions and complex comparisons. Official source: AI Mode
- robots.txt ( Concept or service )
- The file that tells crawlers which URLs on a site they may access, standardized as RFC 9309. Official source: robots.txt
- Gemini ( Concept or service )
- Google's generative AI model family and conversational AI service. Official source: Gemini
- ChatGPT ( Concept or service )
- OpenAI's conversational AI service, which includes a search feature that answers with cited web results. Official source: ChatGPT
- Claude ( Concept or service )
- The conversational AI service made by Anthropic. Official source: Claude
- OpenAI ( Organization )
- The US AI company behind ChatGPT, which operates the GPTBot, OAI-SearchBot and ChatGPT-User crawlers. Official source: OpenAI
- Anthropic ( Organization )
- The US AI company behind Claude, which operates the ClaudeBot, Claude-User and Claude-SearchBot crawlers. Official source: Anthropic
- Perplexity ( Organization )
- The company behind an AI search service that shows its sources; it operates PerplexityBot and Perplexity-User. Official source: Perplexity
- Answer engine optimization (AEO) ( Concept or service )
- Structuring content and information so search engines and AI can present it as the single answer to a question.
- Generative engine optimization (GEO) ( Concept or service )
- The work of making content and brands more visible and more often cited in answers that generative AI builds from multiple sources. Official source: Generative engine optimization (GEO)