AI-bot access checker

What this checks

robots.txt is a plain-text file at the root of your site (yoursite.com/robots.txt) that tells crawlers which paths they're allowed to fetch. For years it only mattered for search engines. Now it's the gatekeeper for the AI crawlers too, and that's the stakes: if your robots.txt quietly blocks the AI bots, you disappear from AI answers. No error, no warning, no ranking drop. You just stop getting cited by ChatGPT and Perplexity, and most teams never notice. This tool reads the rules you paste and tells you, bot by bot, who can reach your content and who's locked out.

The bots that matter

Not all AI crawlers do the same job. The single most important distinction is training vs. search: some bots collect data to train models (you'll never see a referral from them), while others fetch your page live to answer a user's question right now and cite you. Block the wrong one and you either donate your content for free or vanish from the answers that send traffic. Here's the field guide:

Read that table once and the strategy gets obvious: blocking a training bot costs you nothing in search traffic, though blocking Google-Extended also drops you from Gemini grounding; blocking a search bot costs you citations and referrals. The token worth understanding cold is Google-Extended, covered below.

Google-Extended ≠ Google Search

This trips up almost everyone, so it gets its own section. Google-Extended is a robots.txt product token with no crawler of its own. It controls whether content Google crawls from your site may be used to train Gemini models and to ground answers in Gemini Apps and in Grounding with Google Search on Vertex AI. It has no effect on inclusion in Google Search or on ranking. You can Disallow: / for Google-Extended and your pages will still be crawled, indexed, and ranked in ordinary Google Search exactly as before, and they stay eligible for AI Overviews and AI Mode, which follow your Googlebot rules. Anyone who tells you blocking Google-Extended will hurt your rankings is wrong.

How to fix a blocked bot

robots.txt is just grouped User-agent / Allow / Disallow directives. Copy-paste one of these into your robots.txt and adjust.

Allow GPTBot (let ChatGPT train on you):

User-agent: GPTBot
Allow: /

Block AI training, but keep AI search visibility:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Still allow the AI search/browse bots
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

Block every AI crawler (nuclear option):

User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Applebot-Extended
Disallow: /

Note that a missing rule means allowed: robots.txt works by opt-out. If a bot has no group of its own and your User-agent: * group allows crawling, that bot can crawl. To exclude a bot you have to name it explicitly.

Should you block AI crawlers? The trade-off

The right answer depends on your content, so make the trade-off on purpose:

• Block training, allow search. The pragmatic default for most marketing sites. You don't hand your content to model training, but you stay quotable in ChatGPT, Perplexity, and AI Overviews, where the referral traffic lives.
• Allow everything. Maximum reach and citation surface. Sensible if your goal is brand visibility and you're relaxed about your content being in training sets.
• Block everything. Right for paywalled, proprietary, or licensed content you don't want machines reading at all. You opt out of the AI answer engines on that list, while Google Search and AI Overviews still follow your Googlebot rules.

Our AEO-flavoured view: for most businesses, being cited in AI answers is the new front page. Block the training bots if you care about your IP, but think hard before you block the search bots, because that's where discovery is moving.

Limitations to know

The checker reads one robots.txt: the text you paste, or the file it fetches from yourdomain.com/robots.txt when you use Fetch & check. For each bot it uses the group that names that bot, or User-agent: * if none does, and tests the root path "/" only. A rule that blocks one section, such as Disallow: /blog/, will not show up as a block.

A fetch that returns a 4xx status other than 429 counts as no robots.txt, and every bot is shown as allowed, which is how Google's crawlers read it too. A server error, a 429, a timeout or another network failure means the file could not be read: the tool says so and gives no per-bot verdicts until it can read the file. The fetch uses the tool's own user agent, so a server or CDN that answers bots differently may send Google's crawlers a different response.

Readable results can be up to 10 minutes old, so a change you just shipped may take that long to appear. Apart from checking whether /llms.txt exists, the tool reads only robots.txt, so it cannot see blocks set elsewhere, such as firewall or CDN bot rules.

If the AI tools you care about read one, an optional llms.txt file lists your key pages for them. Want a human to audit the whole picture? Grab a free AI SEO audit or see AI SEO services.