Free tool · reads the live file

Does this website block AI crawlers?

Type a domain. The checker reads its robots.txt now and tells you, for each of 34 AI crawlers and opt-out tokens, whether it may crawl, and the exact line that decides.

Also check one page path

Try: · · · ·

What it reads, and how

One request to https://the-website/robots.txt, sent by this site's server because browsers cannot read another site's files. Only that path, only public websites, at most 500 KB, following up to five redirects to another robots.txt. Your browser then reads the file with the rules of RFC 9309:

  • a crawler follows the group that names it, and only that group; with none, the User-agent: * group;
  • the longest matching Allow or Disallow wins, and Allow wins a tie;
  • no file (404) means everything is allowed; a server error means a crawler must stay out.

The crawlers are the ones in the AI crawler directory, each described from its operator's own page. The comparison uses the robots.txt files of the 137 major websites in our weekly study.

Cite this tool

Address: scraperfinder.com/tools/ai-crawler-checker/ Last updated

Scraper Finder. (2026, November 14). AI crawler checker [Online tool]. https://scraperfinder.com/tools/ai-crawler-checker/

Other formats: MLA, Chicago, HTML link, Markdown, BibTeX
MLA
Scraper Finder. "AI crawler checker." Scraper Finder, 14 Nov. 2026, scraperfinder.com/tools/ai-crawler-checker/.
Chicago
Scraper Finder. "AI crawler checker." Last modified November 14, 2026. https://scraperfinder.com/tools/ai-crawler-checker/.
HTML link
<a href="https://scraperfinder.com/tools/ai-crawler-checker/">AI crawler checker</a> (Scraper Finder, updated 14 November 2026)
Markdown
[AI crawler checker](https://scraperfinder.com/tools/ai-crawler-checker/) (Scraper Finder, updated 14 November 2026)
BibTeX
@misc{scraperfinder_toolsaicrawlerchecker_2026,
  author = {{Scraper Finder}},
  title = {AI crawler checker},
  year = {2026},
  howpublished = {\url{https://scraperfinder.com/tools/ai-crawler-checker/}},
  note = {Last updated 14 November 2026}
}

Questions people ask

How does the checker read robots.txt?

It follows RFC 9309, the robots.txt standard: a crawler obeys every group that names it and ignores User-agent: * once it has its own group; the longest matching rule wins and Allow wins a tie. The result shows the group and the line that decided.

What happens if a website has no robots.txt?

A 404 or another 4xx answer means there are no rules, so every crawler may crawl everything. A 5xx error, a 429 or no answer means a crawler following the standard must assume the whole site is off-limits until it can read the file.

Does Allowed mean the AI company will crawl the site?

No. The checker reads what robots.txt asks. A firewall, a bot-management service or a login can still keep crawlers out, and some user-triggered fetchers say they may not read robots.txt at all.

Do you keep the domains I check?

No. The fetch service keeps only daily counts of how many checks ran. Answers are cached for ten minutes, so a repeat check can show the file as it was a few minutes ago.