Agent Web Census

AI crawler blocking on .tv websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 71 are .tv domains and 47 (n) have a readable robots.txt. 10.6% of those 47 block at least one AI crawler by name across the whole site.

Note: small sample (n=47), treat with caution.

AI crawlers blocked by .tv sites with a readable robots.txt (n=47)Applebot-Extended8.5%Amazonbot8.5%GPTBot6.4%ClaudeBot6.4%anthropic-ai6.4%Claude-Web6.4%Google-Extended6.4%CCBot6.4%Bytespider6.4%Meta-ExternalAgent6.4%cohere-ai6.4%Diffbot6.4%omgili6.4%PetalBot4.3%ChatGPT-User2.1%

Percentages are of n = 47 .tv sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.