Agent Web Census

AI crawler blocking on .uk websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 90 are .uk domains and 60 (n) have a readable robots.txt. 36.7% of those 60 block at least one AI crawler by name across the whole site.

Note: small sample (n=60), treat with caution.

AI crawlers blocked by .uk sites with a readable robots.txt (n=60)CCBot30.0%ClaudeBot25.0%GPTBot23.3%PerplexityBot23.3%Meta-ExternalAgent23.3%Applebot-Extended21.7%Bytespider21.7%YouBot21.7%anthropic-ai20.0%Amazonbot18.3%PetalBot18.3%OAI-SearchBot16.7%Claude-Web16.7%Google-Extended16.7%cohere-ai16.7%

Percentages are of n = 60 .uk sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.