Agent Web Census

AI crawler blocking on .es websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 57 are .es domains and 43 (n) have a readable robots.txt. 34.9% of those 43 block at least one AI crawler by name across the whole site.

Note: small sample (n=43), treat with caution.

AI crawlers blocked by .es sites with a readable robots.txt (n=43)GPTBot25.6%ClaudeBot25.6%CCBot25.6%Bytespider25.6%Meta-ExternalAgent23.3%Amazonbot20.9%cohere-ai20.9%Applebot-Extended18.6%Diffbot18.6%omgili18.6%AI2Bot18.6%Timpibot16.3%anthropic-ai14.0%Google-Extended14.0%YouBot14.0%

Percentages are of n = 43 .es sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.