Robots.txt controls whether a crawler may fetch a path. Noindex tells a supporting search engine not to include a page in its index. They solve different problems. If you block a page from crawling, the crawler may not be able to read the page's noindex directive.
Decide what you want to happen
Before editing anything, write the intended result in plain language: this service page should appear in search; this internal search endpoint should not waste crawler requests; this account page should require authentication. Those goals call for different controls. Neither a robots file nor a noindex tag is authentication. Anyone with access to a public URL may still request it, and a robots file can reveal the names of paths you listed.
Test the rule for the exact path
A robots file is scoped to its host and protocol. Start with the same URL you are diagnosing, including the path and query. In the robots checker, select Googlebot and read the matching group and matching rule. A global-looking block can be overridden by a more specific agent group. A longer Allow rule may permit a path beneath a disallowed folder. Do not infer behavior from the first Disallow line you see.
Inspect noindex in both places
Run the noindex checker on the public page. Read the fetched robots/googlebot meta values and the X-Robots-Tag response header. Headers can add restrictions even when a CMS page editor looks correct. If the checker finds a directive, trace its source: an SEO plugin setting, a template, a staging configuration or an edge response rule. Removing the HTML tag will not fix a separate noindex header.
A launch example
Suppose staging used Disallow: / and every page also had a noindex meta tag. Copying staging to production can preserve both. Removing only the robots block lets crawling resume, but the noindex instruction still excludes the pages when read. Removing only noindex can leave crawling blocked. Test each control separately and keep the intentional restrictions on private or staging systems. A broad replace operation across every environment is a poor way to solve a production-specific mistake.
Validate after deployment
Rerun the focused tools after the final deployment. Check a page inside an allowed folder and one inside an intentionally blocked folder, not just the homepage. Save the results with their timestamps. If Google still reports older behavior, compare the last crawl date and its live inspection rather than assuming your change failed. Robots rules can be cached, so your current fetch and a crawler's earlier observation may temporarily differ.
Handle uncertain results carefully
An HTML fallback returned at /robots.txt is a configuration smell. A timeout or server error is also different from a clean missing file. NewSiteCheck marks ambiguous responses for verification rather than calling them a block or a pass. Non-ASCII paths and percent encoding need particular care because rule matching uses URL representation. For those cases, use the authoritative crawler documentation and a rendered/live inspection alongside the simplified tool result.
Illustrative example
User-agent: * Disallow: /drafts/ Allow: /drafts/public-example$ Sitemap: https://example.com/sitemap.xml
Adapt this example to the actual page and site configuration.
A practical action checklist
- State the intended crawl and index behavior separately.
- Test the exact path and the intended user agent.
- Inspect HTML meta and X-Robots-Tag together.
- Keep sensitive content behind access controls.
- Recheck production and compare crawler timestamps.
NewSiteCheck workflow and examples. Review important production decisions against the applicable official documentation.