anthropic-ai — Anthropic's Legacy Crawler
anthropic-ai is a deprecated crawler identifier associated with Anthropic, replaced by ClaudeBot. Learn whether you still need to block it in robots.txt.
QUICK FACTS
anthropic-ai What is anthropic-ai?
anthropic-ai is an older crawler identifier attributed to Anthropic that predates ClaudeBot. It is not listed in Anthropic's current official documentation but is still widely included in publishers' robots.txt files and appears in the community-maintained ai.robots.txt project on GitHub. Listing both anthropic-ai and ClaudeBot in your robots.txt costs nothing and covers older traffic that may still use the legacy identifier.
How to Block anthropic-ai
Add the following to your robots.txt file (located at the root of your website):
User-agent: anthropic-ai Disallow: /
What Happens When You Block anthropic-ai
Blocks legacy Anthropic crawling. Modern crawling is handled by ClaudeBot — block both for complete opt-out from Anthropic training.
Should You Block anthropic-ai?
anthropic-ai is a legacy identifier that has been replaced by newer crawler names. Including it in your robots.txt is a best practice to catch older infrastructure that may still use the original identifier.
anthropic-ai vs Other Anthropic Crawlers
Anthropic operates multiple crawlers, each serving a different purpose:
| User-agent | Purpose | Type |
|---|---|---|
| anthropic-ai | Deprecated training crawler identifier | Legacy / Deprecated |
| ClaudeBot | Collects training data for Claude models | AI Training |
| Claude-User | Live page fetches triggered by Claude users | User-Triggered Fetch |
| Claude-SearchBot | Indexes content for Claude search results | AI Search Index |
Each crawler operates independently. Blocking anthropic-ai does not block ClaudeBot or Claude-User or Claude-SearchBot — you must add a separate rule for each.
GENERATE YOUR ROBOTS.TXT
Use our visual generator to create a robots.txt file that blocks anthropic-ai and any other crawlers you want to opt out of.
RELATED CRAWLERS