Question 1

How does the AI crawler checker work?

Accepted Answer

It fetches your robots.txt and reads it the way crawlers do (RFC 9309): it finds the group that names each crawler, falls back to User-agent: *, and applies the longest matching Allow or Disallow rule to the page you entered. It also looks for an llms.txt file and for noindex in the page's meta tags and X-Robots-Tag header.

Question 2

Should I block GPTBot?

Accepted Answer

Block it if you don't want OpenAI training on your content. It doesn't affect ChatGPT search: that's OAI-SearchBot, a separate crawler. The same split exists at Anthropic (ClaudeBot trains, Claude-SearchBot indexes) and Google (Google-Extended controls Gemini, Googlebot controls Search).

Question 3

If robots.txt allows a crawler, does it actually visit?

Accepted Answer

Not necessarily, and robots.txt is a request, not a lock: some user-triggered fetchers say they don't follow it, and anyone can fake a crawler's user agent. To see real visits, track crawler requests on your server and check them against each provider's published IP ranges. That's what Datasense's AI crawler tracking does.

Question 4

What is llms.txt?

Accepted Answer

A Markdown file at /llms.txt that tells AI assistants what your site is and links to the pages that matter, ideally as clean Markdown. It's a proposed convention (llmstxt.org), not a standard, but it's cheap to add and some assistants and tools read it.

Question 5

Why does my site show as blocked when robots.txt looks fine?

Accepted Answer

If robots.txt answers with a server error (5xx) or doesn't answer at all, RFC 9309 tells crawlers to assume the whole site is off limits. Check that /robots.txt returns 200 or 404, not 500 or a timeout. A firewall or CDN rule blocking crawler user agents can also stop crawlers that robots.txt allows; this check can't see those.

Question 6

Is the checker free?

Accepted Answer

Yes, with no sign-up. It's rate limited to keep it that way. Tracking which crawlers actually visit is part of Datasense Pro.

User agent	Company	What it does
ChatGPT-UserOpenAI	Opens your page when a ChatGPT user asks about it. OpenAI says robots.txt may not apply to these user-initiated visits.
Claude-UserAnthropic	Fetches your page when a Claude user asks a question that needs it.
Perplexity-UserPerplexity	Fetches pages a Perplexity user asked about. Perplexity says it generally ignores robots.txt for these.
Meta-ExternalFetcherMeta	Fetches links for Meta AI when a user shares or asks about them.
DuckAssistBotDuckDuckGo	Pulls the sources behind DuckDuckGo's AI-assisted answers.
MistralAI-UserMistral	Fetches pages for Le Chat users.

User agent	Company	What it does
OAI-SearchBotOpenAI	Decides whether your pages can appear in ChatGPT search results.
Claude-SearchBotAnthropic	Indexes pages so Claude can cite them in search answers.
PerplexityBotPerplexity	Builds the index Perplexity answers and cites from.
GooglebotGoogle	Google Search, including AI Overviews and AI Mode.
BingbotMicrosoft	Bing Search and Microsoft Copilot answers.
ApplebotApple	Siri, Spotlight and Safari suggestions.
DuckDuckBotDuckDuckGo	DuckDuckGo's own search index.

User agent	Company	What it does
GPTBotOpenAI	Collects pages to train OpenAI's models. Blocking it doesn't remove you from ChatGPT search.
ClaudeBotAnthropic	Collects pages to train Anthropic's models.
Google-ExtendedGoogle	A robots.txt token, not a crawler: controls use in Gemini training and grounding. Doesn't affect Google Search.
Applebot-ExtendedApple	A robots.txt token, not a crawler: controls use of Applebot's pages to train Apple's models.
Meta-ExternalAgentMeta	Collects pages to train Meta's AI models.
CCBotCommon Crawl	Builds Common Crawl, an open dataset many models train on.
BytespiderByteDance	ByteDance's crawler, used for its AI models.
AmazonbotAmazon	Amazon's crawler, used for Alexa answers and model training.
cohere-training-data-crawlerCohere	Collects pages to train Cohere's models.
AI2BotAi2	Collects pages for the Allen Institute's open models.

Can AI crawlers read your site?

Answers, search and training are different crawlers

AI answers

AI & search indexing

AI training

Questions people ask

See which AI crawlers actually visit, and what they read