Agent Web Census

AI crawler blocking on .net websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 804 are .net domains and 259 (n) have a readable robots.txt. 22.0% of those 259 block at least one AI crawler by name across the whole site.

AI crawlers blocked by .net sites with a readable robots.txt (n=259)GPTBot15.4%ClaudeBot14.7%CCBot14.3%Bytespider14.3%Google-Extended12.7%PetalBot12.4%Applebot-Extended12.0%Claude-Web11.6%Amazonbot11.6%anthropic-ai11.2%cohere-ai11.2%omgili11.2%Meta-ExternalAgent10.8%Diffbot10.8%PerplexityBot7.7%

Percentages are of n = 259 .net sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.