Agent Web Census

AI crawler blocking on .gov websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 115 are .gov domains and 80 (n) have a readable robots.txt. 15.0% of those 80 block at least one AI crawler by name across the whole site.

Note: small sample (n=80), treat with caution.

AI crawlers blocked by .gov sites with a readable robots.txt (n=80)PetalBot8.8%Bytespider3.8%Amazonbot3.8%ChatGPT-User2.5%ClaudeBot2.5%Claude-User2.5%PerplexityBot2.5%Perplexity-User2.5%CCBot2.5%Meta-ExternalAgent2.5%MistralAI-User2.5%GPTBot1.2%OAI-SearchBot1.2%Claude-SearchBot1.2%anthropic-ai1.2%

Percentages are of n = 80 .gov sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.