NewSiteCheckTest a path, not just a file. See the user-agent group and rule that apply to your URL, including wildcard and end-anchor matching.
No account or email required. Check one public URL at a time.
The tool fetches the site’s robots.txt and evaluates the submitted URL path and query for the selected agent. It combines matching groups at the most specific level, evaluates Allow and Disallow rules, and shows the strongest matching rule. Read the fetched file beside the verdict to understand why a path is allowed or disallowed.
A Disallow result can be intentional for an unimportant crawl path. An allowed path can still carry noindex in its HTML or response headers, so use the noindex checker if you are diagnosing search exclusion. Missing files and uncertain fetch results are handled separately. Robots directives are requests to supporting crawlers, not a way to protect private content from people or every bot.
This is a simplified path-rule inspection, not Google’s live crawl decision. Unusual percent-encoded or non-ASCII equivalence can differ, and crawler caching can preserve older rules. It does not enforce the rules or change your website. Use authentication for sensitive content and compare important Google behavior in Search Console.
Successful checks save a snapshot to your browser workspace. Run the same tool on the same final URL again to compare check statuses with the prior saved result. Export CSV or JSON to keep a file, or print the result as a PDF. Browser data stays on this device and does not run automatic scans.
No. Crawl permission, noindex directives, canonical selection and actual indexing are separate questions.
This tool answers one focused question on a bounded public response. The launch audit combines 40 checks, with a free top-three preview and paid full reports.
No. These checks fetch public information and return evidence. You decide whether to change the site and verify the result after deployment.
Read a practical guide with examples and a step-by-step workflow.