Agent Web Census

AI crawler blocking on .jp websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 199 are .jp domains and 114 (n) have a readable robots.txt. 21.9% of those 114 block at least one AI crawler by name across the whole site.

AI crawlers blocked by .jp sites with a readable robots.txt (n=114)GPTBot14.0%Google-Extended14.0%Applebot-Extended12.3%CCBot12.3%Bytespider12.3%Meta-ExternalAgent12.3%ClaudeBot11.4%anthropic-ai9.6%cohere-ai9.6%ChatGPT-User8.8%OAI-SearchBot8.8%PerplexityBot8.8%Claude-Web7.9%Amazonbot7.9%Diffbot7.0%

Percentages are of n = 114 .jp sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.