robots.txt states what you want crawlers to do. Access logs show what actually hit your server: which user-agent string showed up, which URL it requested, and whether your stack returned 200, 403, or something else. If you are trying to separate training crawlers from search and assistant fetches, or you want proof that GPTBot is getting 403s from a WAF, logs show what actually happened.
The AI Bot Log Analyzer does this. You paste a slice of log text or upload a .log file; parsing runs entirely in your browser. It matches known AI-related user-agents, groups them into assistant/search vs training vs traditional search, and summarizes hits, dates, top paths, and HTTP status codes.
Which user-agents should I look for in access logs?
Vendors publish the strings their crawlers use. Treat those pages as the source of truth when you add firewall allowlists or write robots.txt rules.
OpenAI (OpenAI crawlers documentation) lists agents including GPTBot, OAI-SearchBot, and ChatGPT-User. GPTBot collects training data. OAI-SearchBot indexes pages for ChatGPT search. ChatGPT-User fetches a page when a user asks ChatGPT to.
Anthropic (Anthropic’s crawler FAQ) documents ClaudeBot, Claude-SearchBot, and Claude-User. Older robots.txt files may still name anthropic-ai or claude-web; Anthropic's current article does not list them.
Perplexity (PerplexityBot documentation) documents PerplexityBot and Perplexity-User.
Google does not send a separate “Google-Extended” line in access logs. [Google-Extended](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended) is a robots.txt token for publisher choice about Gemini training and grounding; ordinary Google Search crawling still uses Googlebot.
The analyzer also flags other training-oriented crawlers you may see in real logs (CCBot, Bytespider, Amazonbot, and others) and includes Googlebot and Bingbot for context.
Training crawlers vs search vs user-triggered fetches: why it matters
These categories get lumped together in marketing copy. In logs they are different rows with different user-agent strings.
Training-oriented crawlers (for example GPTBot, ClaudeBot, CCBot) tend to hit many URLs over time. Blocking them in robots.txt is a policy choice many publishers make. That choice does not automatically block every OpenAI or Anthropic fetch.
Retrieval / search crawlers (for example OAI-SearchBot, Claude-SearchBot, PerplexityBot) index pages for search answers and citations in those products. If your goal is to stay visible in AI search, you usually care whether these can fetch HTML successfully, not only whether GPTBot is disallowed.
User-triggered agents (ChatGPT-User, Claude-User, Perplexity-User) often show up when someone pastes a link or asks a product to read a page. You may see fewer hits, but they are a signal that real sessions are pulling your URLs.
robots.txt cannot produce an HTTP 403 by itself. If logs show 403 or 401 for a bot that is allowed in robots.txt, look at CDN bot management, WAF rules, geo blocks, or origin config. The AI Crawler Accessibility Checker checks robots.txt and HTML for a single URL; logs tell you what happened at scale.
How do I run the AI Bot Log Analyzer on my logs?
- Export or copy combined access log lines for the period you care about (a few days is enough to start).
- Open the AI Bot Log Analyzer, paste the text or upload a file.
- Read the summary: total parsed lines, AI bot hits, share of traffic, date range, and whether the file was truncated.
- Sort the bot table by hits or last seen. Check the dominant status column. A training bot with mostly 200s is different from one with mostly 403s.
- Use the timeline for spikes (launch day, robots.txt change, or a CDN rule change).
- Open insights for plain-language flags (for example many 403s to one bot, or 5xx errors wasting crawl attempts).
The tool does not follow individual requests back to your live site. It only reads what you provide.
A worked example (demo sample log)
The analyzer ships a demo sample: synthetic combined-log lines (RFC 5737 documentation IPs, not a real customer host). I ran that sample through the same parser the browser tool uses. Treat it as a teaching example, not traffic from scrawl.tools.
From 28 parsed lines (12–16 Mar 2026), 24 hits matched AI-related bots (~86% of traffic in that slice). Five distinct AI bots appeared: GPTBot (9 hits), ClaudeBot (5), PerplexityBot (4), Bytespider (4), and ChatGPT-User (2). Googlebot and Bingbot also appeared for comparison.
GPTBot hits include 200 responses on content paths but also 403 on /pricing and /api/internal, plus 500 on /docs. The tool’s insights called out GPTBot receiving blocked (4xx) responses and AI bots hitting 5xx errors. That pattern usually means robots.txt might allow the bot, but the edge or app said no. It is a mismatch only logs show.
ChatGPT-User lines include a referer from chat.openai.com, which fits a user asking ChatGPT to open a URL. PerplexityBot and ClaudeBot mostly fetched article and tool paths with 200s.
If this were your site, the next steps would be: confirm robots.txt for OAI-SearchBot and PerplexityBot if you care about AI search, fix the 5xx on /docs, and decide whether 403s on /pricing are intentional before you blame robots.txt.
Where do I pull logs from in practice?
Shared hosting (cPanel, Plesk). “Raw access logs” or “Metrics” often let you download the current access_log. Download a day or a week, not the whole year, to stay under the analyzer’s 5 MB cap.
Nginx / Apache on a VPS. Common paths include /var/log/nginx/access.log or /var/log/apache2/access.log. Rotate copies (.1, .gz) are fine if you decompress first.
Cloudflare. Logpush or instant logs export to S3 or a log vendor. You may need to map Cloudflare’s field names back to combined format, or filter exports to lines that include user-agent.
CDN or WAF in front of origin. Sometimes the edge log is the only place you see bot traffic that never reached your app. If GPTBot is blocked at Cloudflare, origin logs may show nothing while the edge log shows 403.
Strip query strings from paths when you compare to CMS URLs, unless you care about tracking parameters. The analyzer records paths exactly as logged.
How do I combine logs with other checks?
- Policy on robots.txt → Robots.txt Tester or Robots.txt Generator
- Per-URL allow/block and HTML extractability → AI Crawler Accessibility Checker
- Whether Google shows an AI Overview for a query → AI Overview Checker (SERP snapshot, not logs)
- Bulk HTTP status on a URL list after you spot 403/500 paths → Bulk Status Checker
Logs answer “who came and what status did we return?” The crawler checker answers “for this URL, what do we declare and what HTML do we serve?”
What if I see no AI bots at all?
An empty AI section does not always mean bots are absent from the internet. Common causes:
- Log sample too small. A one-hour slice on a low-traffic site may only show Googlebot.
- Bots blocked before logging. Firewall drops may never write a line your app sees.
- Wrong log file. SSL terminator vs origin, or load balancer vs web node.
- User-agent stripped. Rare misconfiguration replaces UA with
-.
Widen the date range or pull edge logs, then re-run. If robots.txt disallows training crawlers and you never see OAI-SearchBot or PerplexityBot, that may match policy. Cross-check with the AI Crawler Accessibility Checker on a money page.
FAQ
Where do I get access logs for AI bot analysis?
From your host or CDN: cPanel raw access logs, Nginx or Apache log files on the server, or log exports from Cloudflare, Fastly, or similar. You need lines that include the user-agent field in combined log format.
What is the difference between GPTBot and ChatGPT-User?
OpenAI documents GPTBot as the crawler for content that may be used to train its generative AI models. ChatGPT-User fetches a page when a user asks ChatGPT to read it. OAI-SearchBot is documented for search-related crawling. See OpenAI’s bot documentation.
Does Google-Extended appear in access logs?
No. It is a robots.txt token. It controls whether Google can use your content for Gemini training and grounding (Gemini Apps, Vertex AI). It does not affect Google Search.
Are my logs sent to Scrawl?
No. Parsing is client-side only.
What log format does the analyzer expect?
Apache or Nginx combined format, up to 5 MB or 250,000 lines per run (truncation is flagged).