ClaudeBot — Anthropic's Training Crawler
ClaudeBot crawls the web to collect training data for Anthropic's Claude models. Learn how to block it in robots.txt without losing visibility in Claude search.
QUICK FACTS
ClaudeBot What is ClaudeBot?
ClaudeBot is Anthropic's production web crawler for gathering training data to build and improve Claude foundation models. It has approximately doubled its crawl rate between Q3 2025 and Q1 2026, suggesting Anthropic is scaling its retrieval infrastructure. Blocking ClaudeBot does not block Claude-User or Claude-SearchBot — each operates independently.
How to Block ClaudeBot
Add the following to your robots.txt file (located at the root of your website):
User-agent: ClaudeBot Disallow: /
What Happens When You Block ClaudeBot
Your future content will be excluded from Claude AI training datasets. Claude-User and Claude-SearchBot are not affected.
Should You Block ClaudeBot?
ClaudeBot is a training crawler — it collects data to build AI models. If you want to prevent your content from being used in future AI training by Anthropic, block it. This is a one-way decision: blocking today only affects future crawls, not data already collected.
ClaudeBot vs Other Anthropic Crawlers
Anthropic operates multiple crawlers, each serving a different purpose:
| User-agent | Purpose | Type |
|---|---|---|
| ClaudeBot | Collects training data for Claude models | AI Training |
| Claude-User | Live page fetches triggered by Claude users | User-Triggered Fetch |
| Claude-SearchBot | Indexes content for Claude search results | AI Search Index |
| anthropic-ai | Deprecated training crawler identifier | Legacy / Deprecated |
Each crawler operates independently. Blocking ClaudeBot does not block Claude-User or Claude-SearchBot or anthropic-ai — you must add a separate rule for each.
GENERATE YOUR ROBOTS.TXT
Use our visual generator to create a robots.txt file that blocks ClaudeBot and any other crawlers you want to opt out of.
RELATED CRAWLERS