Robots.txt Tester

Test and validate your robots.txt. See which crawlers are blocked or allowed, check your rules and sitemap directives, and audit AI bots. Free, no signup.

https://

No credit card required • Instant results

Rule analysis

Parses your Allow / Disallow rules per user-agent against RFC 9309.

Per-bot verdict

See exactly which search and AI crawlers (Googlebot, GPTBot, ClaudeBot…) are blocked.

Sitemap & crawl-delay

Confirms your Sitemap: and Crawl-delay directives are present and well-formed.

Fix-it tips

Actionable recommendations when a rule is missing, conflicting or malformed.

FAQ

Where should robots.txt live?

At the root of your domain - https://example.com/robots.txt. It only applies to the host and scheme it's served from.

Does robots.txt block a page from Google?

Disallow stops crawling, not necessarily indexing. To keep a URL out of search results, use a noindex meta tag or header instead - a disallowed URL can still be indexed if other sites link to it.

Can I block AI crawlers here too?

Yes. This tester evaluates AI bots (GPTBot, ClaudeBot, Google-Extended, PerplexityBot and 20+ more) alongside traditional search crawlers. For a deeper AI-specific report including llms.txt, try our AI Crawler Audit.

This is one check. A full scan runs them all.

Free

The Robots.txt Tester covers one area. A full scan runs every check on the same site in one go, so nothing it does not reach goes unnoticed.

Covered by this check

  • robots.txt RFC 9309 parsing
  • Per-bot Allow / Disallow verdict
  • Sitemap & crawl-delay validation
  • AI crawler rule audit

Added by a full scan

  • SSL certificate validation
  • Security headers analysis
  • Vulnerability scanning
  • DNS & Email security
  • Domain reputation checks
  • Misconfiguration detection
  • Continuous monitoring
  • Email alerts on policy drift
https://

Want it checked automatically? A free account rescans one site every week and emails you new findings. No credit card.

Start free monitoring

More crawler and infrastructure tools

AI Crawler Audit

Audit which AI bots can crawl your site and validate llms.txt.

DNS Lookup

Look up A, AAAA, MX, TXT, NS and other DNS records for any domain.

WHOIS Lookup

Look up domain registration, registrar and nameserver details.

Website Technology Checker

Identify the full tech stack: CMS, frameworks, servers and more.