What is ClaudeBot doing on your site?
ClaudeBot is Anthropic's web crawler. It collects content used to develop Claude. Check whether it has access to your pages and what your robots.txt currently says.
ClaudeBot is Anthropic's primary public crawler. It identifies itself with the User-agent string ClaudeBot and follows robots.txt directives the same way other web crawlers do. If your robots.txt does not name it, it is allowed to crawl by default.
Anthropic also operates other tokens, including anthropic-ai (an older identifier still seen on some sites) and Claude-Web (used when Claude fetches a page during a conversation). A rule that mentions only ClaudeBot does not cover the other two. Pasting your URL above checks all of them, plus 30+ other named AI bots, in one pass.
Frequently asked questions
- What is ClaudeBot? +
- ClaudeBot is the User-agent string Anthropic uses for the crawler that gathers public web content used in developing Claude. It is one of several Anthropic-operated tokens you may see in server logs.
- How is ClaudeBot different from anthropic-ai and Claude-Web? +
- ClaudeBot is the canonical name in Anthropic's current robots.txt guidance. anthropic-ai is an older token still used in some places. Claude-Web is used when Claude fetches a URL during a conversation. The three may need separate robots.txt rules.
- Where does ClaudeBot show up in logs? +
- Server access logs that record User-agent will list ClaudeBot alongside the IP. Anthropic publishes a list of IP ranges so you can verify the request really came from their infrastructure rather than a spoofed UA.
- Does Hyperguard tell me if ClaudeBot is spoofed? +
- The free auditor on this page checks your robots.txt against 33+ named AI bots, including ClaudeBot. Hyperguard's full product detects in real time when a request claims to be ClaudeBot but does not match Anthropic's published IP ranges.
How It Works
Enter your domain
We fetch your public robots.txt — no install, no auth, no site changes.
Diff against AI bot registry
Your rules are checked against 33+ AI crawlers and agents (OpenAI, Anthropic, Perplexity, Google, Meta, ...).
Get a strategic report
Per-bot status, posture flags, and downloadable robots.txt presets you can copy-paste.