Guides

Agent readiness guides

Practical guides to every wall that stops AI agents from reading, citing, and buying from your site. What each one costs you, and the exact fix.

Every guide maps to a check the scanner runs, so you can read the fix, ship it, and verify it in one loop.

Access

GPTBot and robots.txt: allowing OpenAI's crawlers

3 min read

What GPTBot does, what blocking it costs you in ChatGPT answers and referrals, and copy-paste robots.txt examples for all three OpenAI crawlers.

Read the guide →

ClaudeBot, Claude-User, and Claude-SearchBot in robots.txt

3 min read

ClaudeBot, Claude-User, and Claude-SearchBot explained: what each Anthropic crawler does, and the exact robots.txt lines to allow or block each one.

Read the guide →

WAF bot challenges: the wall agents cannot climb

3 min read

Checking-your-browser walls stop GPTBot, ClaudeBot, and shopping agents at the front door. How to allow verified bots in Cloudflare and other WAFs.

Read the guide →

Comprehension

AI agents do not run your JavaScript

3 min read

Most AI crawlers read raw HTML and never execute JavaScript. If your site is a client-side app, agents see a blank shell. Here is how to fix it.

Read the guide →

llms.txt: a map of your site for AI agents

3 min read

llms.txt is a markdown map of your site for AI agents. See the format, a complete example you can adapt, and how to make yours genuinely useful.

Read the guide →

Product schema: hand agents your name, price, and stock

3 min read

Complete Product and Offer JSON-LD with price, availability, returns, and shipping, and why shopping agents skip products they cannot verify.

Read the guide →

Prices agents can read: text and schema, not pixels

3 min read

Agents cannot read prices locked in images or rendered by JavaScript. How to publish prices as text and schema so comparison agents quote you correctly.

Read the guide →

Action

Guest checkout: account walls stop shopping agents cold

3 min read

Why AI agents fail at checkout when you force account creation, and how to enable guest checkout on Shopify, WooCommerce, and custom carts.

Read the guide →

CAPTCHAs and agents: place them where they cost nothing

3 min read

Every CAPTCHA on a key path is a conversion an agent cannot complete. Placement guidance for reCAPTCHA, hCaptcha, and Turnstile that keeps buyers in.

Read the guide →

Trust

Agent endpoints: feeds, OpenAPI, and MCP

4 min read

Product feeds, OpenAPI descriptions, and MCP servers let AI agents integrate with your site instead of scraping it. What to expose first, with examples.

Read the guide →