Robots.txt Tester

Test and validate your robots.txt. See which crawlers are blocked or allowed, check your rules and sitemap directives, and audit AI bots. Free, no signup.

https://

No credit card required • Instant results

Rule analysis

Parses your Allow / Disallow rules per user-agent against RFC 9309.

Per-bot verdict

See exactly which search and AI crawlers (Googlebot, GPTBot, ClaudeBot…) are blocked.

Sitemap & crawl-delay

Confirms your Sitemap: and Crawl-delay directives are present and well-formed.

Fix-it tips

Actionable recommendations when a rule is missing, conflicting or malformed.

FAQ

Where should robots.txt live?

At the root of your domain - https://example.com/robots.txt. It only applies to the host and scheme it's served from.

Does robots.txt block a page from Google?

Disallow stops crawling, not necessarily indexing. To keep a URL out of search results, use a noindex meta tag or header instead - a disallowed URL can still be indexed if other sites link to it.

Can I block AI crawlers here too?

Yes. This tester evaluates AI bots (GPTBot, ClaudeBot, Google-Extended, PerplexityBot and 20+ more) alongside traditional search crawlers. For a deeper AI-specific report including llms.txt, try our AI Crawler Audit.

That's Just the Beginning

FREE

This Robots.txt Tester scan caught some issues. Run a Full Security Scan to uncover hidden threats like exposed secrets, malicious scripts, and supply chain attacks this quick check missed.