Googlebot Simulator: Compare Googlebot vs Browser Fetches

Site owners sometimes discover that Googlebot sees a different page than browsers — a 403 for bots only, a noindex header only on the Googlebot fetch, or a shorter HTML body. Google's URL Inspection tool remains the authority for what Google last crawled. A Googlebot simulator on your own infrastructure is still useful for quick, repeatable checks when you control neither Search Console access nor Google's crawl schedule. The Googlebot Simulator on Scrawl fetches one URL twice from our servers: Googlebot Smartphone user-agent, then desktop Chrome . It also loads robots.txt on the input origin for Googlebot allow/disallow on the path. You get status codes, redirect chains, X-Robots-Tag , meta robots, canonical, title, meta description, H1s, visible word count, link count, and a highlighted diff . It does not execute JavaScript and does not use Google's IP range. What user-agent simulation can and cannot do Can: Reveal whether your origin, CDN, or WAF returns different status codes, headers, or HTML bodies when the User-Agent header changes. Cannot: Prove how Google ranked or indexed the URL, render client-side React/Vue content, or bypass bot management that keys on IP reputation instead of user-agent alone. Treat large diffs as investigation prompts , not accusations. Pair with server logs and URL Inspection. Step-by-step workflow Paste the exact URL you care about (include HTTPS and trailing slash if that is what users hit). Run the simulator and read the cloaking assessment summary first. Open the side-by-side diff — sort mentally by critical rows (status, final URL, X-Robots-Tag , meta robots). Check robots.txt allow/disallow for Googlebot on the path. If diffs are clean but Search Console disagrees, wait for recrawl or use URL Inspection — Google's last fetch may predate your fix. Common causes of user-agent differences | Pattern | Often means | |--------|-------------| | 200…