How to check whether AI search bots can crawl your

A page can look fine in Chrome and still be a thin shell in raw HTML. AI retrieval crawlers mostly read what your server returns without running your JavaScript, and robots.txt may block OAI-SearchBot even when GPTBot is allowed. Before you tweak content for “AI SEO,” it helps to know whether the bots that fetch answers are allowed and whether they would see any text.

The AI Crawler Accessibility Checker takes one public URL. It fetches the page, robots.txt, llms.txt, and llms-full.txt if present, then reports crawler rules, structured data, meta tags, word count in HTML, and a 0–100 score with a prioritized fix list.

What does the checker actually fetch?

Everything runs server-side on Scrawl when you submit a URL:

1. Your page. HTML response, status code, and headers.
2. `/robots.txt` on the same host, parsed into user-agent groups and allow/disallow rules.
3. `/llms.txt` and `/llms-full.txt`, optional markdown discovery files.

The response JSON includes per-agent robots status (allowed, disallowed, or not mentioned), llms.txt structure flags, schema.org types found in JSON-LD, word count and an app-shell warning, title/meta/canonical/h1 checks, an overall score band, and recommendations sorted by impact.

It does not execute JavaScript in a browser. It does not prove a specific vendor will cite you.

How are AI bots grouped in robots.txt?

The checker embeds the same agent list as the API route, aligned with vendor documentation:

| Agent | Vendor | Role in the tool |

| --- | --- | --- |

| GPTBot | OpenAI | Training (OpenAI bots) |

| OAI-SearchBot | OpenAI | Retrieval for search answers |

| ChatGPT-User | OpenAI | User-initiated fetch |

| ClaudeBot | Anthropic | Training (Anthropic FAQ) |

| Claude-SearchBot | Anthropic | Retrieval |

| Claude-User | Anthropic | User-initiated fetch |

| PerplexityBot | Perplexity | Retrieval (PerplexityBot docs) |

| Perplexity-User | Perplexity | User-initiated fetch |

| Google-Extended | Google | Training opt-out token only (Google-Extended) |

ChatGPT-User and Perplexity-User act on a user's request, and their vendors say robots.txt may not apply to them. Use WAF rules if you need a hard block.

Google-Extended is scored as a training opt-out signal. Blocking it does not block Googlebot or normal Google Search crawling.

Deprecated tokens such as anthropic-ai and claude-web are flagged if your robots.txt still mentions them.

Retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) weigh heavily in the score. Blocking them triggers a high-priority “fail” recommendation.

How do I read blocked resources and JS-only warnings?

robots.txt disallowed on a retrieval bot means the checker treats the root path as disallowed for that agent (longest-match rule parsing). Fix the explicit User-agent block or the wildcard * group that is catching them.

403/401 in logs but “allowed” here. Robots and the live edge can disagree. Use the AI Bot Log Analyzer on access logs to see real status codes.

Low word count / app shell. The tool strips scripts and measures visible text in raw HTML. Frameworks that render client-side may score poorly even though Lighthouse looks fine. Server-side render or pre-render the content you want quoted.

Schema. Missing or broken JSON-LD lowers the score. Valid types (Article, FAQPage, BreadcrumbList, etc.) are listed in the response.

llms.txt. Missing file is a warning, not a hard fail. Malformed files (no H1 or links) get a softer warning.

A worked example on scrawl.tools

On 12 Oct 2026 I called the live checker API for https://scrawl.tools/tools/ai-crawler-checker. This is the JSON the tool returns today (your run may differ slightly if robots.txt or the page changes):

• Score: 100/100, band Excellent (robots 30/30, llms 20/20, schema 20/20, content 15/15, meta 15/15).
• Content: 1,343 words in raw HTML; looksLikeAppShell: false.
• Schema: Organization, Person, SoftwareApplication, BreadcrumbList, FAQPage, HowTo (five JSON-LD blocks, zero parse errors).
• llms.txt: present with valid structure; llms-full.txt not present.
• Retrieval bots: OAI-SearchBot and PerplexityBot explicitly allowed in robots.txt; Claude-SearchBot allowed via the * wildcard (not named on its own).
• GPTBot and other training agents: allowed. robots.txt still names deprecated anthropic-ai (allowed).

No recommendations were returned because every weighted check passed. Run the same URL on a staging host behind auth and you will see lower content scores or a failed fetch.

What should I fix first?

1. Unblock retrieval crawlers if they are disallowed and you want AI search visibility.
2. Fix HTTP errors on key templates (5xx and accidental 403s).
3. Put substantive copy in HTML, not only in client bundles.
4. Add accurate JSON-LD for the page type.
5. Publish llms.txt if you want a simple index of key URLs (optional).

For on-page citation structure (headings, FAQ schema, answer-first sections), run the GEO Score Checker. For whether Google shows an AI Overview for a keyword, use the AI Overview Checker.

How is the score weighted?

The API returns a single score out of 100 plus a text band (Excellent, Good, Needs work, Poor). Rough weighting from the code:

• robots.txt / crawler access (~30 points). Disallowed retrieval bots cost the most; disallowed training bots cost less.
• llms.txt (~20 points). Missing file scores zero; present but malformed scores partial credit.
• structured data (~20 points). At least one valid JSON-LD type scores full marks; parse errors score little.
• content accessibility (~15 points). Word count in raw HTML; under 100 words scores zero.
• core meta (~15 points). Title, meta description, h1, and canonical each contribute.

Recommendations are sorted so a blocked OAI-SearchBot or PerplexityBot appears above a missing llms.txt.

Staging, auth, and paywalls

The checker fetches the URL you type. If that URL requires a login, returns a soft 200 with “sign in to read,” or sits behind basic auth, you will see low word counts or error statuses. Test a public URL that matches what a crawler would request.

Subdomains matter. www and bare domain can have different robots.txt files if redirects are misconfigured. Run the checker on the canonical host you publish in sitemaps.

After you change robots.txt

robots.txt changes are not instant in every product, but crawlers will pick them up on later fetches. Re-run the checker on the same URL after you deploy. If you also changed CDN rules, compare with access logs via the AI Bot Log Analyzer so you are not guessing from robots alone.

Pair with Robots.txt Tester when you want Google’s interpretation of the same file for classic search bots. The AI checker uses its own agent list aligned to vendor docs, not only Googlebot.

Who is this check for?

Use it before you spend time rewriting copy for “AI search” when the real blocker is a disallowed retrieval bot or an empty HTML shell. It is also useful after a migration, CDN change, or robots.txt edit when stakeholders ask whether ChatGPT or Perplexity can still read the site. One URL is enough to catch a global Disallow: / mistake.

FAQ

Does this tool browse as ChatGPT or Perplexity?

No. It fetches your URL and robots files directly.

Which bots matter most for AI search?

OAI-SearchBot, Claude-SearchBot, and PerplexityBot for retrieval. Training bots are a separate policy decision.

Will unblocking GPTBot fix ChatGPT search?

Not necessarily. Check OAI-SearchBot and ChatGPT-User separately in OpenAI’s bot documentation.

Why does the tool flag JavaScript-only content?

It only measures raw HTML text. Use SSR or static HTML for critical copy.

Is llms.txt required?

No. It is reported as a nice-to-have discovery file.