Agent Web Census

AI crawler blocking statistics (Nov 2026)

Which AI bots do the world’s most popular websites block in robots.txt? Measured directly on the top 10,000 websites (Tranco list V38GN); scan of 2026-11-18. The crawler scan is repeated weekly (the MCP statistics are updated daily).

23.1%of sites with a readable robots.txt block ≥1 AI crawler by name
15.3%of sites with a readable robots.txt block GPTBot
14.2%of sites with a readable robots.txt block ClaudeBot
9.9%of all 10,000 scanned have llms.txt present (format check)
10,000domains scanned
5,806with a readable robots.txt

Key facts

Cite this: Agent Web Census, “AI crawler blocking statistics”, https://agents.datacensus.org/ai-crawlers/, scan of the top 10,000 websites (Tranco list V38GN) on 2026-11-18.
Definitions

Share of sites blocking each crawler by name (site-wide)

Percent of the 5,806 sites with a readable robots.txt fully blocking each crawlerCCBot16.1%Bytespider15.9%GPTBot15.3%ClaudeBot14.2%Meta-ExternalAgent12.9%Google-Extended12.8%Applebot-Extended12.3%Amazonbot12.2%Diffbot12.1%anthropic-ai11.9%cohere-ai11.8%omgili11.6%PetalBot10.7%Claude-Web9.9%PerplexityBot7.5%FacebookBot6.6%YouBot6.6%ChatGPT-User6.3%Timpibot6.0%ImagesiftBot5.7%AI2Bot4.7%Perplexity-User4.5%DuckAssistBot4.5%OAI-SearchBot4.4%Claude-User4.4%Claude-SearchBot4.1%MistralAI-User3.8%Googlebot0.2%

All crawlers

CrawlerOperatorPurposeBlocked by name (site-wide)Effectively blocked (incl. * rule)PartiallyExplicitly allowed
CCBotCommon Crawltraining16.1%20.2%1.9%1.1%
BytespiderByteDancetraining15.9%20.1%1.3%0.4%
GPTBotOpenAItraining15.3%19.0%3.8%2.5%
ClaudeBotAnthropictraining14.2%18.2%2.9%2.1%
Meta-ExternalAgentMetatraining12.9%17.0%1.5%0.7%
Google-ExtendedGoogletraining12.8%16.6%2.9%2.6%
Applebot-ExtendedAppletraining12.3%16.4%1.4%0.9%
AmazonbotAmazontraining12.2%16.3%1.6%0.8%
DiffbotDiffbottraining12.1%16.4%0.4%0.2%
anthropic-aiAnthropictraining11.9%16.1%1.3%0.8%
cohere-aiCohereuser-triggered11.8%16.1%1.0%0.4%
omgiliWebz.iotraining11.6%15.9%0.3%0.1%
PetalBotHuaweisearch10.7%14.8%0.5%0.2%
Claude-WebAnthropicuser-triggered9.9%14.2%0.9%0.8%
PerplexityBotPerplexitysearch7.5%11.3%3.3%2.7%
FacebookBotMetatraining6.6%10.8%0.8%0.3%
YouBotYou.comsearch6.6%10.9%0.8%0.4%
ChatGPT-UserOpenAIuser-triggered6.3%10.0%3.6%2.6%
TimpibotTimpitraining6.0%10.5%0.2%0.1%
ImagesiftBotHive/Imagesifttraining5.7%10.1%0.2%0.0%
AI2BotAllen Institute for AItraining4.7%9.1%0.4%0.0%
Perplexity-UserPerplexityuser-triggered4.5%8.7%1.7%1.0%
DuckAssistBotDuckDuckGouser-triggered4.5%8.7%1.1%0.3%
OAI-SearchBotOpenAIsearch4.4%8.1%3.6%2.6%
Claude-UserAnthropicuser-triggered4.4%8.4%1.8%1.1%
Claude-SearchBotAnthropicsearch4.1%8.1%2.1%1.0%
MistralAI-UserMistraluser-triggered3.8%8.2%0.6%0.3%
GooglebotGooglesearch0.2%2.9%6.5%2.7%

Percentages are of the 5,806 of 10,000 scanned domains that returned a readable robots.txt. Click a crawler for details.

By popularity rank

BucketDomains scannedReadable robots.txtBlock ≥1 AI botGPTBotClaudeBot
top1001005225.0%21.2%21.2%
top1k1,00051426.8%16.5%16.5%
top10k10,0005,80623.1%15.3%14.2%

Percentages are of the readable-robots.txt count in the same row; buckets are cumulative (top1k includes top100).

By top-level domain

TLDn (readable robots.txt)Block ≥1 AI botGPTBotClaudeBotNote
.com2,98524.4%14.8%14.2%
.net25922.0%15.4%14.7%
.org31020.3%16.1%12.9%
.ru1528.6%7.2%4.6%
.io1039.7%7.8%6.8%
.jp11421.9%14.0%11.4%
.de13530.4%24.4%22.2%
.edu1038.7%5.8%4.9%
.gov8015.0%1.2%2.5%small sample (n=80), treat with caution
.in3920.5%12.8%12.8%small sample (n=39), treat with caution
.cn320.0%0.0%0.0%small sample (n=32), treat with caution
.br6319.0%11.1%9.5%small sample (n=63), treat with caution
.fr7548.0%33.3%33.3%small sample (n=75), treat with caution
.uk6036.7%23.3%25.0%small sample (n=60), treat with caution
.it5844.8%31.0%32.8%small sample (n=58), treat with caution
.tv4710.6%6.4%6.4%small sample (n=47), treat with caution
.co3212.5%9.4%6.2%small sample (n=32), treat with caution
.es4334.9%25.6%25.6%small sample (n=43), treat with caution
.pl4238.1%26.2%9.5%small sample (n=42), treat with caution
.ai3818.4%10.5%7.9%small sample (n=38), treat with caution

n = domains of that TLD with a readable robots.txt; percentages are of n. TLDs with n < 30 are omitted.

Other signals

Method

Scanned the top-N domains of the Tranco list (list ID in tranco_list_id). For each domain we made at most three polite HTTPS GET requests (UA AgentWebCensusBot/0.1): https://<domain>/robots.txt (with one retry on https://www.<domain>/robots.txt only after a connection-level error) and /llms.txt on the host that answered. 10 s timeout, redirects followed, body capped at 500 KB. A robots.txt counts as 'reachable' only if HTTP 200 and the body is non-empty and not HTML (soft-404 HTML pages are treated as invalid/no rules). robots.txt is parsed per RFC 9309 in a simplified way: consecutive User-agent lines form a group; rules of all groups naming the same token are merged; token matching is case-insensitive and exact (no substring/product-version matching); comments, BOM and CRLF are handled. Per bot: blocked_fully (= blocks the whole site) = a group naming the bot has Disallow: / (or /*) and no Allow: / (narrow Allow exceptions such as Allow: /about are tolerated); blocked_partially = it has other non-empty Disallow rules (or Disallow: / overridden by specific Allow rules is NOT counted as fully blocked); allowed_explicitly = a group naming the bot with no effective disallow; not_mentioned = no group names the bot, so it falls back to the '*' group (we record separately how many of those are covered by a '*' block-all). Wildcard patterns, longest-match precedence and Allow/Disallow interplay inside paths are not fully evaluated. Percentages are given both over domains with a reachable robots.txt and over all scanned domains (the latter counts unreachable sites as not blocking). 'AI bots' aggregates exclude Googlebot (comparison only). 'llms.txt present (format check)' means /llms.txt, requested on the host that served robots.txt, returned HTTP 200 with content-type text/markdown or text/plain, a body that starts with '#' (Markdown heading) and is not HTML; content quality is not assessed. Content-Signal counts robots.txt lines starting with 'Content-Signal:'; only key=value pairs with key in search / ai-input / ai-train and value yes / no are parsed (Cloudflare content signals). Caveats: robots.txt is a voluntary request, not enforcement; sites may block by other means (WAF, UA filtering, 403 for our bot) which robots.txt does not show, and some sites serve different robots.txt to different clients or block our scanner (counted as unreachable); network errors and timeouts are counted as unreachable; TLD is the last DNS label (co.uk counts as uk); rank buckets are cumulative (top1k includes top100).

Data: web-latest.json · web-history.csv

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.