What is the difference between ClaudeBot and Claude-User?

ClaudeBot is a bulk crawler gathering content that may improve Anthropic's models. Claude-User is not a crawler at all: it fetches one page because a Claude user asked about it right then. Blocking Claude-User turns away a live, human-initiated request.

Do Anthropic's crawlers respect robots.txt?

Yes. Anthropic documents that its fetchers honor robots.txt directives for their specific tokens as well as wildcard rules, as of mid-2026.

Can I allow Claude to cite me but opt out of training?

Yes. Disallow ClaudeBot only, and keep Claude-SearchBot and Claude-User allowed. The tokens exist separately precisely so sites can make that distinction.

Will allowing these crawlers create meaningful server load?

For a typical site, no; these are polite, low-volume fetchers compared to mainstream search crawlers. If a specific path is expensive to serve, exclude that path inside each group rather than blocking the token outright.

ClaudeBot, Claude-User, and Claude-SearchBot in robots.txt

Updated 2026-07-13 · 3 min read · access pillar

Anthropic, the company behind Claude, operates three distinct fetchers, and your robots.txt can treat each differently. ClaudeBot crawls the public web to gather content that may improve Anthropic's models. Claude-SearchBot crawls to improve the quality of Claude's search results. Claude-User fetches a specific page in real time when a Claude user asks about it.

All three identify themselves with their own user agent token and, per Anthropic's documentation, all three respect robots.txt. Older block lists sometimes reference anthropic-ai and claude-web; as of mid-2026 Anthropic's documentation centers on the three tokens above, so rules should name them explicitly.

The distinction exists so you can be precise. A publisher can welcome the search crawler and the live fetcher while declining the training crawler; a store can welcome all three. What you should not do is let a wildcard rule written years ago make that decision for you.

What a block here actually costs

Claude-User is the one to think hardest about. When it fetches your page, a human is asking Claude about your product at that exact moment. A block returns nothing at the point of maximum intent, and Claude answers from whatever it can read instead: a competitor's page, a reseller's listing, or a stale third-party summary of you.

Claude-SearchBot decides whether you appear in cited search answers at all, and ClaudeBot affects longer-term familiarity with your content. Blocking all three with one rule turns off every one of those surfaces at once. The traffic you save is a rounding error; the answers you drop out of are where buying decisions increasingly start.

Allow Anthropic's crawlers (copy-paste)

Robots rules are per token: a crawler follows the most specific group that names it and ignores the rest. Add all three groups so a future blanket rule cannot silently catch them, and check the live file first; a missing robots.txt already passes by default, while an inherited wildcard block fails everything at once.

text

# robots.txt: allow Anthropic crawling, search, and live user fetches
User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

Keep sensitive paths excluded per group (for example Disallow: /account) rather than blocking the whole token.

Opting out of training while staying quotable

If your concern is model training, block ClaudeBot alone and leave the search and user-fetch tokens open. You keep citations and live lookups, the surfaces that carry buyers, while expressing the training preference.

One warning: robots.txt only governs crawlers that choose to read it. If your CDN or WAF also has an AI-bot blocking toggle enabled, that block wins regardless of what robots.txt says. Check both layers; the WAF guide below covers the second one.

text

# Training opt-out only: block ClaudeBot, keep answers and lookups
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

How AgentReady checks it

The scanner asks: does robots.txt allow Claude's crawlers? It evaluates all three tokens against your robots.txt, including wildcard rules. An explicit or blanket block fails; a missing robots.txt passes by default. The check is worth 3 points in the Access pillar.

The methodology's reasoning: Claude's crawlers (ClaudeBot, Claude-User, Claude-SearchBot) gather the content Claude quotes to buyers. A block here removes you from those answers.

Frequently asked questions

What is the difference between ClaudeBot and Claude-User?: ClaudeBot is a bulk crawler gathering content that may improve Anthropic's models. Claude-User is not a crawler at all: it fetches one page because a Claude user asked about it right then. Blocking Claude-User turns away a live, human-initiated request.
Do Anthropic's crawlers respect robots.txt?: Yes. Anthropic documents that its fetchers honor robots.txt directives for their specific tokens as well as wildcard rules, as of mid-2026.
Can I allow Claude to cite me but opt out of training?: Yes. Disallow ClaudeBot only, and keep Claude-SearchBot and Claude-User allowed. The tokens exist separately precisely so sites can make that distinction.
Will allowing these crawlers create meaningful server load?: For a typical site, no; these are polite, low-volume fetchers compared to mainstream search crawlers. If a specific path is expensive to serve, exclude that path inside each group rather than blocking the token outright.