Agent Web Census

AI crawler blocking on .ru websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 295 are .ru domains and 152 (n) have a readable robots.txt. 8.6% of those 152 block at least one AI crawler by name across the whole site.

AI crawlers blocked by .ru sites with a readable robots.txt (n=152)GPTBot7.2%CCBot5.3%Bytespider5.3%ClaudeBot4.6%PetalBot3.9%ChatGPT-User3.3%Amazonbot3.3%FacebookBot3.3%cohere-ai3.3%ImagesiftBot3.3%omgili3.3%anthropic-ai2.6%Google-Extended2.6%Meta-ExternalAgent2.6%Applebot-Extended2.0%

Percentages are of n = 152 .ru sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.