AI crawler blocking on .gov websites (Nov 2026)
Among the top 10,000 websites (scan of 2026-11-18), 115 are .gov domains and 80 (n) have a readable robots.txt. 15.0% of those 80 block at least one AI crawler by name across the whole site.
Definitions
- Blocked site-wide (by name): a robots.txt group naming the bot has
Disallow: /withoutAllow: /(narrow Allow exceptions are tolerated). - Effectively blocked: blocked site-wide as above, or a blanket
User-agent: *group withDisallow: /. - Partially restricted: a group naming the bot has other Disallow rules.
- Purpose labels (training, search, user-triggered) follow each operator’s public documentation.