Perplexity and Claude Search Visibility

Which bot must be allowed to appear in Perplexity?

You need to allow PerplexityBot. Perplexity states that this bot is for showing and linking sites in search results and is not used to collect content for training AI foundation models.

Perplexity also has an agent called Perplexity-User. It visits pages in order to answer a user's question, and Perplexity states that because this is a user-requested fetch, it generally ignores robots.txt rules.

After changing settings, allow some time before checking. Perplexity says robots.txt changes can take up to 24 hours to take effect, and sites that use a web application firewall (WAF) may need to add the bot to an allowlist.

Which bots must be allowed to appear in Claude search?

The bots related to search visibility are Claude-SearchBot and Claude-User, and their purpose differs from that of ClaudeBot, which handles collection for model training.

According to Anthropic's official guidance, Claude-SearchBot navigates and indexes the web to improve search result quality, and blocking it may reduce your visibility and accuracy in search results. Claude-User accesses websites when a user asks Claude a question, and blocking it prevents content from being retrieved in response to user requests, which may reduce visibility.

ClaudeBot collects web content that could contribute to model training, and blocking it signals that the content should be excluded from future training data. Anthropic states that its bots honor robots.txt and also support Crawl-delay. If you do not want collection for training but do want search visibility, you can consider writing separate rules for each bot.

Why might a site not appear even after allowing the bots?

Blocking settings outside robots.txt may still be in place, the bots may not be able to read the body content, or the content may be in a form that is hard to use as an answer to a question.

Above all, allowing bots is only a condition that makes the site readable. The specific criteria for which pages are used in answers are not public, and no one can guarantee AI citations.

• Bot-blocking settings in a firewall, CDN or hosting service sometimes take effect before robots.txt and block AI crawlers.
• Some AI crawlers may not run JavaScript, so it is safer to put important body content in the HTML the server sends.
• robots.txt changes may not have taken effect yet (up to 24 hours for Perplexity).
• If a page does not answer the question directly or lacks evidence, it may be hard for it to be chosen as a source.

How should content be prepared?

Attaching verifiable evidence such as sources, quotations and statistics is supported by research, and the basic direction is to add an answer-first structure on top of that.

The paper that proposed the term generative engine optimization (GEO) (Aggarwal et al., KDD 2024) reported that adding citations to sources, adding quotations and adding statistics were the most effective methods, and it also validated these methods on Perplexity (Perplexity.ai). Keyword stuffing did not work well. However, these were experiments in a research setting and generative engines keep changing, so the same figures are not guaranteed in real services.

This direction is no different from the basics of search engine optimization (SEO) and answer engine optimization (AEO). With text content that crawlers can read, question-style subheadings and paragraphs that state the conclusion first, you can address Google and Naver search and AI search at the same time.