AI crawler blocking on .uk websites (Nov 2026)
Among the top 10,000 websites (scan of 2026-11-18), 90 are .uk domains and 60 (n) have a readable robots.txt. 36.7% of those 60 block at least one AI crawler by name across the whole site.
Definitions
- Blocked site-wide (by name): a robots.txt group naming the bot has
Disallow: /withoutAllow: /(narrow Allow exceptions are tolerated). - Effectively blocked: blocked site-wide as above, or a blanket
User-agent: *group withDisallow: /. - Partially restricted: a group naming the bot has other Disallow rules.
- Purpose labels (training, search, user-triggered) follow each operator’s public documentation.