Agent Web Census

AI crawler blocking on .com websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 5,060 are .com domains and 2,985 (n) have a readable robots.txt. 24.4% of those 2,985 block at least one AI crawler by name across the whole site.

AI crawlers blocked by .com sites with a readable robots.txt (n=2985)Bytespider16.6%CCBot16.4%GPTBot14.8%ClaudeBot14.2%Meta-ExternalAgent13.2%Applebot-Extended12.9%Amazonbot12.6%Google-Extended12.5%anthropic-ai12.4%Diffbot12.4%omgili11.9%cohere-ai11.5%PetalBot11.1%Claude-Web9.6%PerplexityBot7.4%

Percentages are of n = 2,985 .com sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.