Agent Web Census

AI crawler blocking on .it websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 83 are .it domains and 58 (n) have a readable robots.txt. 44.8% of those 58 block at least one AI crawler by name across the whole site.

Note: small sample (n=58), treat with caution.

AI crawlers blocked by .it sites with a readable robots.txt (n=58)CCBot37.9%Bytespider36.2%ClaudeBot32.8%GPTBot31.0%Google-Extended27.6%PerplexityBot27.6%cohere-ai27.6%Diffbot25.9%Amazonbot24.1%anthropic-ai22.4%Claude-Web22.4%DuckAssistBot22.4%Applebot-Extended20.7%YouBot20.7%ImagesiftBot20.7%

Percentages are of n = 58 .it sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.