AI crawler blocking statistics (Nov 2026)
23.1%of sites with a readable robots.txt block ≥1 AI crawler by name
15.3%of sites with a readable robots.txt block GPTBot
14.2%of sites with a readable robots.txt block ClaudeBot
9.9%of all 10,000 scanned have llms.txt present (format check)
10,000domains scanned
5,806with a readable robots.txt
Key facts
- As of 2026-11-18, 23.1% of the 5,806 top-10,000 sites with a readable robots.txt (13.4% of all 10,000 scanned) block at least one AI crawler by name across the whole site.
- As of 2026-11-18, 15.3% of the 5,806 top-10,000 sites with a readable robots.txt (8.9% of all 10,000 scanned) block OpenAI’s GPTBot site-wide and 14.2% (8.2% of all 10,000 scanned) block Anthropic’s ClaudeBot, versus 0.2% for Googlebot (search, not an AI-training crawler; shown for comparison).
- As of 2026-11-18, 10.0% of the 5,806 top-10,000 sites with a readable robots.txt (5.8% of all 10,000 scanned) block all four of GPTBot, ClaudeBot, Google-Extended and CCBot site-wide.
- As of 2026-11-18, 9.9% of all 10,000 scanned top sites have an llms.txt present (format check: HTTP 200, text/plain or text/markdown, starts with a Markdown heading, not HTML; requested on the host that served robots.txt).
- As of 2026-11-18, the most-blocked AI crawler of the 5,806 top-10,000 sites with a readable robots.txt is CCBot (16.1% block it site-wide).
- As of 2026-11-18, 6.8% of the 5,806 top-10,000 sites with a readable robots.txt (3.9% of all 10,000 scanned) contain a Cloudflare Content-Signal directive.
Cite this: Agent Web Census, “AI crawler blocking statistics”, https://agents.datacensus.org/ai-crawlers/, scan of the top 10,000 websites (Tranco list V38GN) on 2026-11-18.
Definitions
- Blocked site-wide (by name): a robots.txt group naming the bot has
Disallow: /withoutAllow: /(narrow Allow exceptions are tolerated). - Effectively blocked: blocked site-wide as above, or a blanket
User-agent: *group withDisallow: /. - Partially restricted: a group naming the bot has other Disallow rules.
- Purpose labels (training, search, user-triggered) follow each operator’s public documentation.
- Googlebot (search, not an AI-training crawler) is shown for comparison.
Share of sites blocking each crawler by name (site-wide)
All crawlers
| Crawler | Operator | Purpose | Blocked by name (site-wide) | Effectively blocked (incl. * rule) | Partially | Explicitly allowed |
|---|---|---|---|---|---|---|
| CCBot | Common Crawl | training | 16.1% | 20.2% | 1.9% | 1.1% |
| Bytespider | ByteDance | training | 15.9% | 20.1% | 1.3% | 0.4% |
| GPTBot | OpenAI | training | 15.3% | 19.0% | 3.8% | 2.5% |
| ClaudeBot | Anthropic | training | 14.2% | 18.2% | 2.9% | 2.1% |
| Meta-ExternalAgent | Meta | training | 12.9% | 17.0% | 1.5% | 0.7% |
| Google-Extended | training | 12.8% | 16.6% | 2.9% | 2.6% | |
| Applebot-Extended | Apple | training | 12.3% | 16.4% | 1.4% | 0.9% |
| Amazonbot | Amazon | training | 12.2% | 16.3% | 1.6% | 0.8% |
| Diffbot | Diffbot | training | 12.1% | 16.4% | 0.4% | 0.2% |
| anthropic-ai | Anthropic | training | 11.9% | 16.1% | 1.3% | 0.8% |
| cohere-ai | Cohere | user-triggered | 11.8% | 16.1% | 1.0% | 0.4% |
| omgili | Webz.io | training | 11.6% | 15.9% | 0.3% | 0.1% |
| PetalBot | Huawei | search | 10.7% | 14.8% | 0.5% | 0.2% |
| Claude-Web | Anthropic | user-triggered | 9.9% | 14.2% | 0.9% | 0.8% |
| PerplexityBot | Perplexity | search | 7.5% | 11.3% | 3.3% | 2.7% |
| FacebookBot | Meta | training | 6.6% | 10.8% | 0.8% | 0.3% |
| YouBot | You.com | search | 6.6% | 10.9% | 0.8% | 0.4% |
| ChatGPT-User | OpenAI | user-triggered | 6.3% | 10.0% | 3.6% | 2.6% |
| Timpibot | Timpi | training | 6.0% | 10.5% | 0.2% | 0.1% |
| ImagesiftBot | Hive/Imagesift | training | 5.7% | 10.1% | 0.2% | 0.0% |
| AI2Bot | Allen Institute for AI | training | 4.7% | 9.1% | 0.4% | 0.0% |
| Perplexity-User | Perplexity | user-triggered | 4.5% | 8.7% | 1.7% | 1.0% |
| DuckAssistBot | DuckDuckGo | user-triggered | 4.5% | 8.7% | 1.1% | 0.3% |
| OAI-SearchBot | OpenAI | search | 4.4% | 8.1% | 3.6% | 2.6% |
| Claude-User | Anthropic | user-triggered | 4.4% | 8.4% | 1.8% | 1.1% |
| Claude-SearchBot | Anthropic | search | 4.1% | 8.1% | 2.1% | 1.0% |
| MistralAI-User | Mistral | user-triggered | 3.8% | 8.2% | 0.6% | 0.3% |
| Googlebot | search | 0.2% | 2.9% | 6.5% | 2.7% |
By popularity rank
| Bucket | Domains scanned | Readable robots.txt | Block ≥1 AI bot | GPTBot | ClaudeBot |
|---|---|---|---|---|---|
| top100 | 100 | 52 | 25.0% | 21.2% | 21.2% |
| top1k | 1,000 | 514 | 26.8% | 16.5% | 16.5% |
| top10k | 10,000 | 5,806 | 23.1% | 15.3% | 14.2% |
By top-level domain
| TLD | n (readable robots.txt) | Block ≥1 AI bot | GPTBot | ClaudeBot | Note |
|---|---|---|---|---|---|
| .com | 2,985 | 24.4% | 14.8% | 14.2% | |
| .net | 259 | 22.0% | 15.4% | 14.7% | |
| .org | 310 | 20.3% | 16.1% | 12.9% | |
| .ru | 152 | 8.6% | 7.2% | 4.6% | |
| .io | 103 | 9.7% | 7.8% | 6.8% | |
| .jp | 114 | 21.9% | 14.0% | 11.4% | |
| .de | 135 | 30.4% | 24.4% | 22.2% | |
| .edu | 103 | 8.7% | 5.8% | 4.9% | |
| .gov | 80 | 15.0% | 1.2% | 2.5% | small sample (n=80), treat with caution |
| .in | 39 | 20.5% | 12.8% | 12.8% | small sample (n=39), treat with caution |
| .cn | 32 | 0.0% | 0.0% | 0.0% | small sample (n=32), treat with caution |
| .br | 63 | 19.0% | 11.1% | 9.5% | small sample (n=63), treat with caution |
| .fr | 75 | 48.0% | 33.3% | 33.3% | small sample (n=75), treat with caution |
| .uk | 60 | 36.7% | 23.3% | 25.0% | small sample (n=60), treat with caution |
| .it | 58 | 44.8% | 31.0% | 32.8% | small sample (n=58), treat with caution |
| .tv | 47 | 10.6% | 6.4% | 6.4% | small sample (n=47), treat with caution |
| .co | 32 | 12.5% | 9.4% | 6.2% | small sample (n=32), treat with caution |
| .es | 43 | 34.9% | 25.6% | 25.6% | small sample (n=43), treat with caution |
| .pl | 42 | 38.1% | 26.2% | 9.5% | small sample (n=42), treat with caution |
| .ai | 38 | 18.4% | 10.5% | 7.9% | small sample (n=38), treat with caution |
Other signals
- 4.6% of the 5,806 top-10,000 sites with a readable robots.txt block all bots via
User-agent: */Disallow: /. - A robots.txt names on average 3.169 of the AI crawlers we track (10.555 among files that name any).
- llms.txt present (format check): 9.9% of all 10,000 scanned
Method
Data: web-latest.json · web-history.csv