Agent Web Census

AI crawler blocking on .io websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 213 are .io domains and 103 (n) have a readable robots.txt. 9.7% of those 103 block at least one AI crawler by name across the whole site.

AI crawlers blocked by .io sites with a readable robots.txt (n=103)GPTBot7.8%Bytespider7.8%ClaudeBot6.8%CCBot6.8%Amazonbot6.8%PetalBot6.8%anthropic-ai5.8%Google-Extended5.8%Applebot-Extended5.8%Meta-ExternalAgent5.8%cohere-ai5.8%Diffbot5.8%omgili5.8%Claude-Web4.9%ImagesiftBot1.0%

Percentages are of n = 103 .io sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.