Claude-SearchBot — Anthropic's Search Indexer
Claude-SearchBot indexes web pages for Claude's search feature. Learn how it works and how to control it with robots.txt.
QUICK FACTS
Claude-SearchBot What is Claude-SearchBot?
Claude-SearchBot crawls and indexes web content to improve the quality of search results inside Claude. It does not collect data for model training. Blocking Claude-SearchBot may reduce the accuracy and visibility of your content when Claude users perform searches.
How to Block Claude-SearchBot
Add the following to your robots.txt file (located at the root of your website):
User-agent: Claude-SearchBot Disallow: /
What Happens When You Block Claude-SearchBot
Your content will not be indexed for Claude's search feature. Training (ClaudeBot) and user-triggered fetching (Claude-User) are unaffected.
Should You Block Claude-SearchBot?
Claude-SearchBot is a search indexer — it indexes your content so it can appear in Anthropic's AI-powered search results. Blocking it removes your visibility in that product but does not affect traditional search engines (Google, Bing). Consider whether AI search traffic is valuable to you before blocking.
Claude-SearchBot vs Other Anthropic Crawlers
Anthropic operates multiple crawlers, each serving a different purpose:
| User-agent | Purpose | Type |
|---|---|---|
| Claude-SearchBot | Indexes content for Claude search results | AI Search Index |
| ClaudeBot | Collects training data for Claude models | AI Training |
| Claude-User | Live page fetches triggered by Claude users | User-Triggered Fetch |
| anthropic-ai | Deprecated training crawler identifier | Legacy / Deprecated |
Each crawler operates independently. Blocking Claude-SearchBot does not block ClaudeBot or Claude-User or anthropic-ai — you must add a separate rule for each.
GENERATE YOUR ROBOTS.TXT
Use our visual generator to create a robots.txt file that blocks Claude-SearchBot and any other crawlers you want to opt out of.
RELATED CRAWLERS