AI crawler blocking on .ru websites (Nov 2026)
Among the top 10,000 websites (scan of 2026-11-18), 295 are .ru domains and 152 (n) have a readable robots.txt. 8.6% of those 152 block at least one AI crawler by name across the whole site.
Definitions
- Blocked site-wide (by name): a robots.txt group naming the bot has
Disallow: /withoutAllow: /(narrow Allow exceptions are tolerated). - Effectively blocked: blocked site-wide as above, or a blanket
User-agent: *group withDisallow: /. - Partially restricted: a group naming the bot has other Disallow rules.
- Purpose labels (training, search, user-triggered) follow each operator’s public documentation.