GPTBot and robots.txt: allowing OpenAI's crawlers

GPTBot is OpenAI's web crawler. It identifies itself with the user agent token GPTBot (the full string includes GPTBot/1.2 and a link to OpenAI's documentation), fetches publicly reachable pages, and respects robots.txt. What it collects may be used to train and improve OpenAI's models, which is why it is the token most robots.txt files single out.

GPTBot is not the only OpenAI fetcher, and that matters for your robots.txt. OAI-SearchBot crawls to build the search index that ChatGPT search cites, and ChatGPT-User fetches a page on demand when a user asks ChatGPT about it. Each has its own token, and a rule written for GPTBot does not apply to the other two.

Why blocking GPTBot costs you revenue

When ChatGPT answers a buying question, the products and vendors it names come from pages its crawlers could read. If your robots.txt blocks OpenAI's crawlers, the comparison still happens; it just happens without you. The customer sees two competitors quoted with prices and links, and your absence reads as irrelevance, not principle.

The block is invisible from the inside. Your analytics still show human traffic, your pages still rank in classic search, and nothing errors. The only symptom is that assistant-driven citations, referrals, and agent-initiated checkouts go to sites that let the crawlers in. Many sites are blocking by accident: a robots.txt copied from a template, or a CDN bot toggle someone enabled during an attack and never revisited.

Allow OpenAI's crawlers (copy-paste)

robots.txt lives at the root of your domain, for example https://yourstore.com/robots.txt. Crawlers read the group that matches their token most specifically; if a group names GPTBot, GPTBot follows that group and ignores the wildcard group. That means an explicit Allow group wins even when a blanket User-agent: * Disallow: / exists above it.

Audit before you edit: open your live robots.txt in a browser and search it for GPTBot and for User-agent: *. Sites are regularly surprised by what a theme, a plugin, or a years-old template put there. If the file does not exist at all, that is a pass by default; crawlers treat a missing robots.txt as no restrictions.

# robots.txt: allow OpenAI crawling, search, and live user fetches
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Add these groups to your existing robots.txt. Keep any Disallow lines for /cart, /admin, or /account inside each group if you have private paths.

Blocking deliberately, without losing citations

Blocking GPTBot is a legitimate choice if you do not want your content used for model training. Make it a decision, not an accident, and keep it narrow: GPTBot is the training-related crawler, while OAI-SearchBot feeds the search results that cite and link you. Blocking all three tokens to express a training preference also removes you from answers that would have sent buyers.

# Opt out of training crawls, stay visible in ChatGPT search
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

OpenAI documents each token's purpose and publishes IP ranges to verify genuine traffic (platform.openai.com/docs/bots).

How AgentReady checks it

The scanner fetches your robots.txt and asks one question: does robots.txt allow GPTBot? An explicit GPTBot block fails, a blanket wildcard block fails, and a missing robots.txt passes by default because crawlers treat no file as no restrictions. The check is worth 3 points in the Access pillar.

The methodology's reasoning: GPTBot fetches the pages ChatGPT cites and shops from. If robots.txt blocks it, you are invisible to the largest assistant audience and the referrals it sends.