Agent Web Census

AI crawler blocking on .ai websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 51 are .ai domains and 38 (n) have a readable robots.txt. 18.4% of those 38 block at least one AI crawler by name across the whole site.

Note: small sample (n=38), treat with caution.

AI crawlers blocked by .ai sites with a readable robots.txt (n=38)PetalBot13.2%GPTBot10.5%Google-Extended10.5%Applebot-Extended10.5%Amazonbot10.5%ClaudeBot7.9%Bytespider5.3%ChatGPT-User2.6%OAI-SearchBot2.6%anthropic-ai2.6%Claude-Web2.6%PerplexityBot2.6%CCBot2.6%Meta-ExternalAgent2.6%cohere-ai2.6%

Percentages are of n = 38 .ai sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.