AI Crawler Audit

See which AI bots can crawl your site: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 20+ more. Free, no signup.

https://

No credit card required • Instant results

Per-bot verdict

25+ documented AI crawlers checked against your robots.txt.

llms.txt

Validates structure against the llmstxt.org spec.

Meta tags

Detects noai and noimageai directives.

Fix-it tips

Actionable recommendations specific to your site.

FAQ

Does blocking AI crawlers hurt SEO?

No. Google-Extended is separate from Googlebot, and OAI-SearchBot is separate from regular search indexing. Block AI training bots without affecting your search ranking.

What's llms.txt?

A markdown summary file at /llms.txt (see llmstxt.org) that helps AI assistants understand your site's structure.

Do crawlers actually respect robots.txt?

The major published bots (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Applebot-Extended) have publicly committed to honour it. For lesser-known scrapers, combine with Cloudflare AI bot protection or ASN-level blocking.

This is one check. A full scan runs them all.

Free

The AI Crawler Audit covers one area. A full scan runs every check on the same site in one go, so nothing it does not reach goes unnoticed.

Covered by this check

  • Per-bot verdict for 25+ AI crawlers
  • robots.txt RFC 9309 parsing
  • llms.txt spec validation
  • noai / noimageai meta tag check

Added by a full scan

  • SSL certificate validation
  • Security headers analysis
  • Vulnerability scanning
  • DNS & Email security
  • Domain reputation checks
  • Misconfiguration detection
  • Continuous monitoring
  • Email alerts on policy drift
https://

Want it checked automatically? A free account rescans one site every week and emails you new findings. No credit card.

Start free monitoring

More crawler and infrastructure tools

Robots.txt Tester

Test robots.txt rules, including AI crawler directives.

DNS Lookup

Look up A, AAAA, MX, TXT, NS and other DNS records for any domain.

Website Technology Checker

Identify the full tech stack: CMS, frameworks, servers and more.

WHOIS Lookup

Look up domain registration, registrar and nameserver details.