Agent Web Census

AI crawler blocking on .in websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 106 are .in domains and 39 (n) have a readable robots.txt. 20.5% of those 39 block at least one AI crawler by name across the whole site.

Note: small sample (n=39), treat with caution.

AI crawlers blocked by .in sites with a readable robots.txt (n=39)Bytespider17.9%CCBot15.4%Meta-ExternalAgent15.4%cohere-ai15.4%GPTBot12.8%ClaudeBot12.8%Google-Extended12.8%Diffbot12.8%omgili12.8%PetalBot12.8%Claude-Web10.3%Amazonbot10.3%anthropic-ai7.7%Applebot-Extended7.7%DuckAssistBot7.7%

Percentages are of n = 39 .in sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.