Agent Web Census

AI crawler blocking on .fr websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 97 are .fr domains and 75 (n) have a readable robots.txt. 48.0% of those 75 block at least one AI crawler by name across the whole site.

Note: small sample (n=75), treat with caution.

AI crawlers blocked by .fr sites with a readable robots.txt (n=75)CCBot38.7%GPTBot33.3%ClaudeBot33.3%Google-Extended33.3%Bytespider30.7%Amazonbot30.7%cohere-ai30.7%anthropic-ai29.3%Claude-Web26.7%YouBot26.7%omgili25.3%ChatGPT-User24.0%PerplexityBot24.0%Applebot-Extended22.7%FacebookBot22.7%

Percentages are of n = 75 .fr sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.