Agent Web Census

AI crawler blocking on .de websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 177 are .de domains and 135 (n) have a readable robots.txt. 30.4% of those 135 block at least one AI crawler by name across the whole site.

AI crawlers blocked by .de sites with a readable robots.txt (n=135)CCBot25.2%GPTBot24.4%ClaudeBot22.2%Bytespider22.2%Applebot-Extended21.5%Google-Extended19.3%Meta-ExternalAgent18.5%Diffbot17.8%omgili14.8%PerplexityBot11.9%FacebookBot11.9%cohere-ai11.1%Amazonbot10.4%anthropic-ai8.9%PetalBot8.9%

Percentages are of n = 135 .de sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.