Agent Web Census

AI crawler blocking on .cn websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 104 are .cn domains and 32 (n) have a readable robots.txt. 0.0% of those 32 block at least one AI crawler by name across the whole site.

Note: small sample (n=32), treat with caution.

AI crawlers blocked by .cn sites with a readable robots.txt (n=32)GPTBot0.0%ChatGPT-User0.0%OAI-SearchBot0.0%ClaudeBot0.0%Claude-User0.0%Claude-SearchBot0.0%anthropic-ai0.0%Claude-Web0.0%Google-Extended0.0%Applebot-Extended0.0%PerplexityBot0.0%Perplexity-User0.0%CCBot0.0%Bytespider0.0%Amazonbot0.0%

Percentages are of n = 32 .cn sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.