Robots.txt Checker

Check your robots.txt file — analyze Allow/Disallow rules, Sitemap and User-agents

Bot to evaluate

Up to 6 checks/IP/hour plus shared limits. Only robots.txt is fetched, not the target page. 500 KiB, 5 redirects; private networks are blocked.

May this bot crawl the URL under robots.txt?

Enter the full URL including path/query, select a bot and check. Matching bot groups are combined; the most specific group and longest matching rule decide. Allow wins ties. Compare the reported group, rule and line against the raw file.

This models current rules, not actual bot access, indexing, ranking or AI citations. Training differs from search. ChatGPT-User requests may not follow robots.txt. Google-Extended is a product-control token, not a separate HTTP crawler. llms.txt does not guarantee citations.

Example: Disallow: /private with Allow: /private/open in the same group permits /private/open. Check query strings too: $ anchors the end of the complete path/query.

Sources and method: RFC 9309 · Google · OpenAI · Anthropic

What does robots.txt do for SEO?

Robots.txt requests which paths compliant crawlers may fetch. Inspect the exact User-agent group and path before changing a rule.

A Disallow rule controls crawling, not guaranteed removal from search. The checker retrieves robots.txt, lists User-agent groups and rules, and shows declared sitemaps and raw text. Review a rule in its group context: a path under one bot group does not automatically apply to every crawler. To exclude a public page from indexing, Google must be able to read its noindex instruction. Confidential content needs authentication. A missing file and a server error are different situations; review the HTTP response when troubleshooting access.

Counts Allow/Disallow rules and User-agents
Detects Sitemap declarations in robots.txt
Shows the raw content for review

Get started free

Example: interpret a finding and fix it

Illustration only, not a check of your website.

After a staging release, the production file contains "User-agent: *" and "Disallow: /". Verify the environment and affected bot group before removing the rule; preserve restrictions that remain intentional.

To exclude a public page from search, allow Google to crawl its noindex instruction. Blocking the URL in robots.txt can prevent Google from reading noindex. A blocked URL can still appear when linked from elsewhere.

Robots.txt is not access control. Account pages and confidential documents require server-side authentication even when a Disallow rule exists.

Google Search Central

How to use in 3 steps

1Enter the complete URL including path/query and select a bot.
2Check robots.txt, then review the deciding group, rule, line and HTTP status.
3Review the Allow/Disallow rules, User-agents and Sitemap declaration.

Frequently asked questions

How is robots.txt different from a noindex tag?

robots.txt blocks crawling (Google never reads the page), while noindex allows crawling but keeps the page out of the index. To remove a page from Google, use noindex, not robots.txt.

How do I know if robots.txt is blocking by mistake?

Check for a "Disallow: /" line (blocks everything) or blocks on important folders like /blog or /products. The tool clearly lists the blocked paths.

Should I declare the Sitemap in robots.txt?

Yes. Adding "Sitemap: https://yoursite.com/sitemap.xml" helps every search engine find your sitemap without manual submission.

What if my site has no robots.txt?

That is fine — search engines may then crawl everything. Still, creating a robots.txt lets you control crawling and declare your sitemap.