Agent Web Census

AI crawler blocking on .co websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 60 are .co domains and 32 (n) have a readable robots.txt. 12.5% of those 32 block at least one AI crawler by name across the whole site.

Note: small sample (n=32), treat with caution.

AI crawlers blocked by .co sites with a readable robots.txt (n=32)GPTBot9.4%Google-Extended9.4%Bytespider9.4%ClaudeBot6.2%Applebot-Extended6.2%CCBot6.2%Amazonbot6.2%Meta-ExternalAgent6.2%cohere-ai6.2%Diffbot6.2%omgili6.2%PetalBot6.2%anthropic-ai3.1%Claude-Web3.1%FacebookBot3.1%

Percentages are of n = 32 .co sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.