Agent Web Census

AI crawler blocking on .org websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 431 are .org domains and 310 (n) have a readable robots.txt. 20.3% of those 310 block at least one AI crawler by name across the whole site.

AI crawlers blocked by .org sites with a readable robots.txt (n=310)GPTBot16.1%CCBot16.1%Bytespider13.9%Google-Extended13.2%ClaudeBot12.9%Meta-ExternalAgent12.9%Applebot-Extended12.3%anthropic-ai11.9%Amazonbot11.3%cohere-ai11.3%Diffbot11.0%omgili10.6%Claude-Web8.7%PetalBot8.7%ImagesiftBot4.8%

Percentages are of n = 310 .org sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.