Check robots.txt and Sitemaps Without Confusing Crawl and Indexing

robots.txt controls compliant crawling, while a sitemap helps disclose URLs. Neither is a guarantee of indexing. Check the requested hostname, the exact URL and indexing directives separately when troubleshooting a missing page.

Review the crawl rules

Fetch the robots.txt for the intended protocol and host. Identify the user-agent group and path rules relevant to the page. Compare allowed and disallowed paths carefully; a rule affecting a directory may have a much wider scope than the page you are investigating. Do not publish credentials or private records just because a path is disallowed.

Review the sitemap as an inventory

Check that the response is the expected XML rather than an HTML error or login page. Follow sitemap-index entries when applicable. Compare representative URLs with their final response, preferred canonical and intended indexing status. A parser reporting a valid file has not necessarily crawled every URL inside it.

Worked example: the sitemap includes a blocked article

Imagine an article is listed in the sitemap but its directory is disallowed. If you want it indexed, review whether that crawl restriction is intentional, then verify the page response and indexing directives. If you want the article excluded, use an appropriate indexing control that the crawler can read; simply blocking crawling may leave a previously discovered URL eligible to appear without a useful snippet. This is an illustrative configuration conflict.

Retest the intended outcome

1. Save the rule or sitemap change and its reason.
2. Fetch the files again from the production host.
3. Test the specific affected path and an unaffected path.
4. Compare the delivered robots meta and X-Robots-Tag.
5. Follow the URL's status in Search Console without assuming immediate recrawling.

Use Robots Checker and Sitemap Checker as separate checks. Do not remove broad rules without checking their purpose or promise that resubmitting a sitemap will make Google index all entries.

See Google's robots.txt introduction and sitemap overview.