Reference and study · updated
34 AI crawlers and opt-out tokens, each described from its operator's own page: what it collects, and what the operator says about robots.txt.
We read the robots.txt files of 137 major websites the way RFC 9309 tells a crawler to. 50 of them (36%) block at least one AI training crawler from every page. GPTBot is blocked by 38 (28%) and ClaudeBot by 37 (27%); 25 block GPTBot but let OpenAI's search crawler in. 81 block none of them (59%).
Check any website's robots.txtBuild robots.txt rulesDownload the CSVCite
Each bar counts the websites whose robots.txt keeps the crawler off every page: by naming it in a group of its own (dark), or only through the rules for every crawler, User-agent: * (yellow). Out of 137 websites whose file could be read.
"Every page" means the home page and any other page are disallowed. Websites that close only some paths to a crawler are counted in the CSV, with the crawler-by-crawler verdict for each website.
The name to write after User-agent:, who runs it, what it is for and what its operator says about robots.txt, read on 14 November 2026. Follow the source link before you rely on a line: operators change these pages.
Collect pages that may train AI models. These are the ones most robots.txt rules are written for.
Crawls pages that may be used to train OpenAI's generative AI foundation models.
robots.txt: OpenAI: disallowing GPTBot says the site's content should not be used to train its foundation models.
38 of 137 major websites block it from every page; 47 name it.
Collects web content that may be used to train and develop Anthropic's models.
robots.txt: Anthropic: its bots honour robots.txt directives.
37 of 137 major websites block it from every page; 44 name it.
Crawls the web for training AI models and for improving products by indexing content.
robots.txt: Meta documents blocking it with robots.txt disallow rules.
26 of 137 major websites block it from every page; 29 name it.
Improves Amazon's products and services. Amazon says what it collects may be used to train Amazon AI models.
robots.txt: Amazon: honours the user-agent and allow/disallow rules in robots.txt.
28 of 137 major websites block it from every page; 25 name it.
Collects web content for datasets that train Mistral's generative AI models.
robots.txt: Mistral: webmasters can disallow it in robots.txt.
17 of 137 major websites block it from every page; 3 name it.
Discovers web content used to train the Allen Institute's open language models.
robots.txt: The operator's page does not say how it treats robots.txt.
23 of 137 major websites block it from every page; 10 name it.
No crawler sends these names. They are switches in robots.txt that decide whether pages a company's normal crawler collects may train its AI models.
A robots.txt token, not a crawler: it decides whether pages Google crawls may be used to train future Gemini models. It has no effect on Google Search.
robots.txt: Google: crawling is done with Google's usual user agents; Google-Extended only reads as a robots.txt group.
28 of 137 major websites block it from every page; 35 name it.
A robots.txt token, not a crawler: it decides whether pages Applebot collects may train Apple's foundation models. Blocking it keeps pages in Apple's search features.
robots.txt: Apple: Applebot-Extended does not crawl; disallow it to opt out of model training.
29 of 137 major websites block it from every page; 26 name it.
Index pages so an AI search engine or assistant can find, cite and link them. Blocking them can take a site out of those answers.
Finds and surfaces websites in ChatGPT search results.
robots.txt: OpenAI: sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers.
13 of 137 major websites block it from every page; 32 name it.
Indexes pages to improve the quality of Claude's search results.
robots.txt: Anthropic: its bots honour robots.txt directives.
14 of 137 major websites block it from every page; 24 name it.
Surfaces and links websites in Perplexity search results. Perplexity says it is not used to crawl for AI model training.
robots.txt: Perplexity: allow it in robots.txt for your site to appear in its search results.
21 of 137 major websites block it from every page; 35 name it.
Crawls the web to improve the quality of Meta AI search results.
robots.txt: Meta: allowing it in robots.txt helps Meta cite and link your content.
15 of 137 major websites block it from every page; 8 name it.
Improves search experiences in Amazon products. Amazon says it does not crawl for generative AI training.
robots.txt: Amazon: honours the user-agent and allow/disallow rules in robots.txt.
14 of 137 major websites block it from every page; 3 name it.
Indexes pages for Mistral's search. Mistral says the content is not used for AI training of any kind.
robots.txt: Mistral's page says nothing more about robots.txt for this agent.
14 of 137 major websites block it from every page; 1 name it.
Fetches pages in real time for DuckDuckGo's AI-assisted answers, which cite their sources. DuckDuckGo says the data never trains AI models.
robots.txt: DuckDuckGo: respects robots.txt; a change takes effect within 72 hours.
15 of 137 major websites block it from every page; 11 name it.
Discovers and indexes pages for You.com search results.
robots.txt: You.com: fully respects robots.txt and caches it for 30 minutes.
21 of 137 major websites block it from every page; 10 name it.
Fetch one page because a person asked an assistant about it. Several operators say robots.txt may not apply to them.
Visits a page when a person's request in ChatGPT or a custom GPT needs it.
robots.txt: OpenAI: because a user starts these visits, robots.txt rules may not apply.
21 of 137 major websites block it from every page; 37 name it.
Fetches a page when a Claude user asks something that needs it.
robots.txt: Anthropic: its bots honour robots.txt directives, this one included.
17 of 137 major websites block it from every page; 24 name it.
Visits a page when a Perplexity user's question needs it.
robots.txt: Perplexity: because a user asked for the page, this fetcher generally ignores robots.txt rules.
18 of 137 major websites block it from every page; 16 name it.
Fetches individual links a user asks for, for Meta's agentic AI features.
robots.txt: Meta: because the fetches are user requested, it may bypass robots.txt rules.
16 of 137 major websites block it from every page; 12 name it.
Supports user actions, such as Alexa answers that need current information. Amazon says it does not crawl for generative AI training.
robots.txt: Amazon: honours the user-agent and allow/disallow rules in robots.txt.
13 of 137 major websites block it from every page; 2 name it.
Visits a page when a user of Mistral's assistant asks for it. Not an automatic crawler.
robots.txt: Mistral: robots.txt decides which sites these user requests may reach.
16 of 137 major websites block it from every page; 4 name it.
Build web collections that other companies and researchers download.
Builds Common Crawl, an open repository of web crawl data that anyone can download and analyse.
robots.txt: Common Crawl: CCBot respects robots.txt.
34 of 137 major websites block it from every page; 36 name it.
The crawler behind Webz.io, a web data service used by media monitors and research institutes.
robots.txt: Webz.io: says it always complies with robots.txt requests.
29 of 137 major websites block it from every page; 20 name it.
AI-related agents with a narrower job.
Checks the safety of web pages submitted as ads on ChatGPT.
robots.txt: OpenAI's page gives no robots.txt guidance for it.
15 of 137 major websites block it from every page; 4 name it.
Crawls sites whose owners asked for it, to build Vertex AI Agents. No effect on Google Search.
robots.txt: Listed among Google's common crawlers, which follow robots.txt.
20 of 137 major websites block it from every page; 10 name it.
The name of img2dataset, an open-source program that downloads large image collections for AI datasets. Anyone can run it; it is not one company's crawler.
robots.txt: No operator page.
21 of 137 major websites block it from every page; 8 name it.
No operator page
Older names that still fill many robots.txt files. Their operators no longer list them.
An older name many robots.txt files still carry. It is not on Anthropic's current crawler page.
robots.txt: Not on the operator's current page.
26 of 137 major websites block it from every page; 23 name it.
An older name many robots.txt files still carry. It is not on Anthropic's current crawler page.
robots.txt: Not on the operator's current page.
24 of 137 major websites block it from every page; 17 name it.
An older Meta name many robots.txt files still carry. It is not on Meta's current crawler page.
robots.txt: Not on the operator's current page.
24 of 137 major websites block it from every page; 16 name it.
An older name of Webz.io's crawler. Webz.io's own post gives the agent as omgili.
robots.txt: Not on the operator's current page.
24 of 137 major websites block it from every page; 15 name it.
Names found in robots.txt files whose operator publishes no page about them.
A crawler third parties attribute to ByteDance. ByteDance publishes no page about it.
robots.txt: No operator statement found.
40 of 137 major websites block it from every page; 33 name it.
No operator page
A name many block lists carry. Cohere's crawler page does not list it and says Cohere does not crawl to train generative AI models at this time.
robots.txt: Cohere's page names no active agent and says its crawlers must respect robots.txt.
24 of 137 major websites block it from every page; 16 name it.
The crawler of Diffbot, which sells web extraction and a knowledge graph. Diffbot's documentation has no crawler page.
robots.txt: No operator statement found.
26 of 137 major websites block it from every page; 18 name it.
Tick what you want to keep out. The rules update as you go; paste them into your robots.txt above or below your other groups.
# Tick a crawler or pick a starting point.
One group lists every ticked name and closes the whole site to them. A crawler with its own group stops reading your User-agent: * rules, so anything you still want it to follow belongs in this group too. User-triggered fetchers may ignore the file; see each entry above.
User-agent: * groups; the longest matching rule wins and Allow wins a tie. "Every page" means the home page and an ordinary deep page are both disallowed. The same reader runs in the live checker and is tested against shared cases.What a crawler actually does. robots.txt states a wish; it cannot show firewalls, bot-management rules, paywalls or crawlers that ignore it. A website whose robots.txt could not be read is left out of the counts, never guessed.
One row per website, one column per crawler: blocked, blocked-except, partial or allowed, with +named when the website's own group for that crawler decides. The JSON adds the directory and the totals.
Address: scraperfinder.com/ai-crawlers/ Last updated Licence: CC BY 4.0, free to reuse with a credit and a link
Scraper Finder. (2026, November 14). AI crawlers: every robots.txt name, and the major websites that block them [Data set]. https://scraperfinder.com/ai-crawlers/
Scraper Finder. "AI crawlers: every robots.txt name, and the major websites that block them." Scraper Finder, 14 Nov. 2026, scraperfinder.com/ai-crawlers/.
Scraper Finder. "AI crawlers: every robots.txt name, and the major websites that block them." Last modified November 14, 2026. https://scraperfinder.com/ai-crawlers/.
<a href="https://scraperfinder.com/ai-crawlers/">AI crawlers: every robots.txt name, and the major websites that block them</a> (Scraper Finder, updated 14 November 2026)
[AI crawlers: every robots.txt name, and the major websites that block them](https://scraperfinder.com/ai-crawlers/) (Scraper Finder, updated 14 November 2026)
@misc{scraperfinder_aicrawlers_2026,
author = {{Scraper Finder}},
title = {AI crawlers: every robots.txt name, and the major websites that block them},
year = {2026},
howpublished = {\url{https://scraperfinder.com/ai-crawlers/}},
note = {Last updated 14 November 2026}
}Not by itself. OpenAI documents three agents: GPTBot collects training data, OAI-SearchBot finds pages for ChatGPT search, and ChatGPT-User visits a page when a person asks. Blocking GPTBot asks OpenAI not to train on your pages; your pages can still appear in ChatGPT search unless OAI-SearchBot is blocked too.
No. Google says Google-Extended has no user agent string of its own: Google's usual crawlers fetch the pages, and the Google-Extended group in robots.txt only decides whether they may be used to train Gemini models. It has no effect on Google Search. Applebot-Extended works the same way for Apple.
robots.txt is a request, not a lock. Most operators listed here say their automatic crawlers honour it; OpenAI, Perplexity and Meta say their user-triggered fetchers may not. Blocking for certain takes a firewall or bot-management rule, which robots.txt cannot show.
Under RFC 9309 a crawler obeys only the group that names it. If robots.txt has a GPTBot group, GPTBot ignores everything under User-agent: *. A site that blocks everyone with User-agent: * but adds an empty GPTBot group has let GPTBot in.
The robots.txt files are read again every week. Operator descriptions are re-checked when an operator changes its page; each entry links to the page it comes from. The date at the top moves only when a number changes.
