Agent Web Census

AI crawler blocking on .edu websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 125 are .edu domains and 103 (n) have a readable robots.txt. 8.7% of those 103 block at least one AI crawler by name across the whole site.

AI crawlers blocked by .edu sites with a readable robots.txt (n=103)GPTBot5.8%Bytespider5.8%ClaudeBot4.9%anthropic-ai4.9%Google-Extended4.9%CCBot4.9%Claude-Web3.9%Applebot-Extended3.9%Amazonbot3.9%cohere-ai3.9%Diffbot3.9%omgili3.9%PetalBot3.9%Meta-ExternalAgent2.9%PerplexityBot1.9%

Percentages are of n = 103 .edu sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.