Google Robots.txt Tester
Test any URL against your robots.txt rules. Simulate Googlebot, Bingbot, and other crawlers to verify allow and disallow directives. Free SEO tool, no login.
- Fetch robots.txt
- Test a URL path
- Adjust blocking rules
- Browser-based — no install
- Instant results
This Google robots.txt tester simulates Googlebot against your live robots.txt file. Enter your site URL and a page path to see whether crawlers are allowed or blocked — and which rule matched.
Robots.txt is a plain-text file at your site root that tells crawlers which URL paths they are allowed to fetch . It is a crawl gate, not an index remote. Google may still show a URL in results—sometimes without a description—if other sites link to it, even when robots.txt blocks Googlebot from downloading the page. That distinction trips up more migrations than bad redirects. Teams Disallow a folder to "hide" staging, then wonder why Search Console still lists those URLs. Or they leave everything crawlable and expect noindex inside blocked HTML to work—it cannot, because Googlebot never sees the tag. This guide explains how Allow , Disallow , and wildcards match paths, how crawl differs from index, mistakes that look fine in a text editor, how AI crawler user-agents fit in, and a practical testing workflow . When you need to paste a file and probe paths quickly, use the Robots.txt Tester once you know what you are checking. Crawl vs index (and where robots.txt sits) Crawling is a bot requesting a URL and receiving a response. Indexing is the search engine deciding whether to store that URL and show it in results. Robots.txt only affects the first step for bots that follow the file. | Signal | What it controls | Typical location | |--------|------------------|------------------| | robots.txt Disallow | Whether the bot may fetch the URL | /robots.txt on the host | | noindex | Whether the URL should appear in search results | HTML meta or X-Robots-Tag header | | Canonical | Which URL you prefer indexed when duplicates exist | HTML link or header | | HTTP 404 / 410 | The URL is gone | Server response | Google's documentation is explicit: blocking with robots.txt is not a substitute for noindex if your goal is to…
Read the full guideWhat this tester returns
Fetch your live robots.txt from the site URL, then test a path (for example `/blog/post-1`) with a user-agent such as Googlebot or `*`. The JSON response includes **allowed** (true/false), **matchedRule**, **matchedDirective** (allow, disallow, or default-allow), and which **user-agent group** applied. If robots.txt is missing (404), paths default to allowed.
Allow vs Disallow precedence (how Scrawl matches)
For the selected user-agent group, every Allow and Disallow line is tested against the path you enter. The **longest matching rule wins**. If two rules tie on length, **Allow beats Disallow**. An empty `Disallow:` line means “allow all” and is ignored as a block.
Wildcards: `*` in a rule becomes “any characters” when matching (Google-style). A trailing `$` anchors the end of the path. Example file:
``` User-agent: * Disallow: /search Allow: /search/help Disallow: /admin/ ```
| Path you test | Result | Winning rule | | --- | --- | --- | | `/search/results` | Blocked | `Disallow: /search` (longer than any Allow that does not match) | | `/search/help` | Allowed | `Allow: /search/help` (longer than `Disallow: /search`) | | `/admin/login` | Blocked | `Disallow: /admin/` | | `/blog/hello` | Allowed | No disallow match → default allow |
How to read the UI
**Allowed + matched Allow** — crawling is permitted for that path under the simulated bot. **Blocked + matched Disallow** — the crawler should not fetch the URL; fix the rule before shipping if unintended. **default-allow** with no matched rule — either no robots.txt or no rule covered that path.
Common mistakes
- Staging `Disallow: /` copied to production — test `/` and your top templates after every deploy. - Broad `Disallow: /search` blocking faceted URLs you still want indexed — add a longer `Allow` for the paths that should stay crawlable. - Expecting robots.txt to remove URLs from Google’s index — disallow stops fetching; use noindex on a crawlable URL when you need de-indexing. - Testing only the homepage — also test paginated paths, `/wp-admin/`, and CDN asset paths if you added rules for them.
Frequently Asked Questions
How do I test my robots.txt file?
Enter your site's robots.txt URL and the specific page URL you want to test, then select the crawler to simulate. The tool fetches your live robots.txt, parses every rule, and tells you whether the page is allowed or blocked — and which exact rule matched the decision.
What happens if robots.txt blocks Googlebot?
If Googlebot is blocked from crawling a URL, that page cannot be indexed and will not appear in Google Search results. A common disaster is deploying a staging robots.txt (which typically blocks all crawlers) to production. Check immediately after any server change by testing your homepage and key pages.
What is the difference between robots.txt disallow and noindex?
Disallow in robots.txt prevents crawlers from fetching a URL but does not remove it from the index — Google may still index the URL if it sees links to it. A noindex meta tag (or X-Robots-Tag header) must be read by Googlebot to take effect, which requires the page to be crawlable in the first place.
Can robots.txt block specific crawlers?
Yes. Robots.txt uses user-agent directives to apply rules to specific crawlers. You can block Googlebot while allowing Bingbot, or block all crawlers with User-agent: * while creating exceptions for specific bots. The Robots.txt Tester lets you simulate any user-agent to see which rules apply.
Does robots.txt affect PageRank?
Disallowing a URL in robots.txt prevents Googlebot from crawling it, which means PageRank from links to that URL is not passed further to other pages it links to. If you want to prevent indexing without losing internal PageRank flow, use a noindex directive on a crawlable page rather than a robots.txt disallow.
Compare with SmallSEOTools — free, no sign-up
Related tools
Indexability Checker: Can Google Index This Page?
Crawl & AuditCheck whether one page can be indexed from server signals: HTTP status, robots.t…
Google Index Checker
Technical SEOCheck if any URL is indexed by Google instantly. Find why pages are not indexed…
Noindex Checker
Technical SEOCheck meta robots and X-Robots-Tag headers across multiple URLs at once. Find no…
Robots.txt Generator
Utility ToolsCreate robots.txt files with advanced rules to exercise fine-grained control ove…