Free AI crawler checker
See whether ChatGPT, Claude, Perplexity, Google AI Overviews and Copilot can read your site. We check your robots.txt and test your live site as each crawler, so a firewall block doesn't slip through.
- Cost
- 100% free
- Checks
- 15 AI crawlers
- Not needed
We read your robots.txt and load your homepage once as each crawler. Nothing is stored.
See what comes back
An example for a site that blocks AI training on purpose, and has a firewall quietly turning Perplexity away. Yours shows the same checks for your site.
AI search can see you
5 of 6
Assistants can open your page
2 of 3
Training crawlers allowed
4 of 6
Fix first
Perplexity answers can't see acme.com, so you can't appear in its answers. Use the robots.txt below and check your firewall's bot settings.
AI search and answers
These decide whether you can be found and cited in AI answers. Block them and you disappear from those answers.
- Allowed
OAI-SearchBot · ChatGPT search
Some private paths are blocked, which is normal.
- Allowed
Claude-SearchBot · Claude search
Some private paths are blocked, which is normal.
- Turned away by firewall
PerplexityBot · Perplexity answers
Perplexity won't surface or link you in its answers.
- Allowed
Googlebot · Google Search and AI Overviews
Some private paths are blocked, which is normal.
- Allowed
Bingbot · Bing and Microsoft Copilot
Some private paths are blocked, which is normal.
- Allowed
Applebot · Siri, Spotlight and Apple Intelligence
Some private paths are blocked, which is normal.
Fetches when someone asks
These open your page when a person asks an assistant about it. Blocking them stops the assistant reading you live.
- Allowed
ChatGPT-User · ChatGPT reading a page for a user
Some private paths are blocked, which is normal.
- Allowed
Claude-User · Claude reading a page for a user
Some private paths are blocked, which is normal.
- Turned away by firewall
Perplexity-User · Perplexity reading a page for a user
Perplexity can't open your page when someone asks about it.
AI model training
These collect content to train future models. Blocking them is a choice and doesn't remove you from AI answers.
- Blocked in robots.txt
GPTBot · OpenAI model training
Your content isn't used to train OpenAI models. ChatGPT search is unaffected.
- Allowed
ClaudeBot · Anthropic model training
Some private paths are blocked, which is normal.
- Allowed
Google-Extended · Gemini training and grounding
Some private paths are blocked, which is normal. A control setting, not a crawler: only robots.txt applies.
- Allowed
Applebot-Extended · Apple model training
Some private paths are blocked, which is normal. A control setting, not a crawler: only robots.txt applies.
- Blocked in robots.txt
CCBot · Common Crawl (used by many AI models)
You leave the open dataset many AI models are trained on.
- Allowed
meta-externalagent · Meta AI training
Some private paths are blocked, which is normal.
Your robots.txt allows these, but your firewall doesn't
A request using the name of PerplexityBot, Perplexity-User was turned away, while a normal browser got through. That usually means a content delivery network (CDN) or firewall bot rule (Cloudflare, Vercel, Akamai, a WordPress security plugin). Look for “AI bots” or “bot fight” settings and allow the search crawlers. Some firewalls check the crawler's IP address and let the real one through, so confirm in your firewall's logs.
Your AI-ready robots.txt
Add this to your robots.txt. It keeps the private paths you already block as it is.
# AI search and answers: allowed, so assistants can find and cite you User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User Allow: / Disallow: /admin/ # AI model training: blocked (doesn't affect AI search or Google) User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: meta-externalagent Disallow: / Sitemap: https://acme.com/sitemap.xml
Googlebot, Bingbot and Applebot are left to your existing rules. Blocking training doesn't affect AI search, Google Search or AI Overviews.
Your current robots.txt
User-agent: * Disallow: /admin/ User-agent: GPTBot User-agent: CCBot Disallow: /
How the AI Crawler Checker works
- 01
Enter your website
Type your domain, for example acme.com. No sign-up and no email.
- 02
We read your robots.txt
Each AI crawler is matched to the rules that apply to it, the way the crawlers themselves read them, following the robots.txt standard (RFC 9309).
- 03
We test your live site
Your homepage is loaded once as each crawler and once as a normal browser, to catch firewall and content delivery network (CDN) blocks that robots.txt doesn't show.
- 04
Read your result
See which AI search engines, assistants and training crawlers can read you, grouped by what blocking them means.
- 05
Copy the fix
Copy an AI-ready robots.txt that allows AI search, lets you choose on training, and keeps your private paths blocked.
Built and maintained by Swapan Kumar Manna, go-to-market and AI search strategist. Updated .
Questions about the AI Crawler Checker
What it does, what it costs, and how far to trust the result.
01What is an AI crawler checker?
A free tool that tells you whether AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot can read your website, by checking your robots.txt and testing your live site.
02Which AI crawlers decide whether I appear in AI answers?
The search crawlers: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Googlebot for Google AI Overviews and AI Mode, Bingbot for Microsoft Copilot, and Applebot for Siri and Apple Intelligence. Block these and you can't be cited.
03Does blocking GPTBot or Google-Extended remove me from AI search?
No. GPTBot, ClaudeBot and Google-Extended control AI model training. OpenAI, Anthropic and Google each say blocking them doesn't affect their search features or Google Search. Many sites block training and allow search.
04Why test the live site, not just robots.txt?
Firewalls and content delivery networks (CDNs) often turn AI crawlers away even when robots.txt allows them. The checker loads your homepage as each crawler and as a normal browser, so you see blocks robots.txt alone would hide.
05Is the AI crawler checker free?
Yes, completely free, with no sign-up and no email. Nothing about your site is stored.
06What should I do after the check?
Copy the suggested robots.txt and check your firewall's bot settings if anything was turned away. Being readable is step one; being clearly positioned is what gets you recommended.
Try another tool
llms.txt Generator
Free llms.txt generator and validator. Builds an llms.txt from your own pages and sitemap, checked against the official spec, ready to edit, copy or download.
- About 1 minute
- Reads your live site
- 100% free · no email needed
Ideal Customer Profile Generator
Free AI ideal customer profile (ICP) generator. Say what you sell and get three ideal customer profiles, ranked by fit: company size, who decides, buying triggers, who to skip and a LinkedIn search to find them.
- About 2 minutes
- AI-assisted
- 100% free · all 3 profiles with your email
Buyer Persona Generator
Free AI buyer persona generator for businesses (B2B) and consumers (B2C). Add your website and what you sell, and get up to three buyer personas: goals, pain points, buying triggers, objections and the channels they trust.
- About 3 minutes
- AI-assisted
- 100% free · full personas with your email