AI Crawler Audit

See which AI bots can crawl your site: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 20+ more. Free, no signup.

https://

No credit card required • Instant results

Per-bot verdict

25+ documented AI crawlers checked against your robots.txt.

llms.txt

Validates structure against the llmstxt.org spec.

Meta tags

Detects noai and noimageai directives.

Fix-it tips

Actionable recommendations specific to your site.

FAQ

Does blocking AI crawlers hurt SEO?

No. Google-Extended is separate from Googlebot, and OAI-SearchBot is separate from regular search indexing. Block AI training bots without affecting your search ranking.

What's llms.txt?

A markdown summary file at /llms.txt (see llmstxt.org) that helps AI assistants understand your site's structure.

Do crawlers actually respect robots.txt?

The major published bots (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, Applebot-Extended) have publicly committed to honour it. For lesser-known scrapers, combine with Cloudflare AI bot protection or ASN-level blocking.

More crawler and infrastructure tools

Robots.txt Tester

Test robots.txt rules, including AI crawler directives.

DNS Lookup

Look up A, AAAA, MX, TXT, NS and other DNS records for any domain.

Website Technology Checker

Identify the full tech stack: CMS, frameworks, servers and more.

WHOIS Lookup

Look up domain registration, registrar and nameserver details.

This is one layer of the scan

Free

This AI Crawler Audit scan covers one category. A full scan adds the checks it does not reach: exposed secrets, malicious scripts, and supply chain risks in your third-party code.