Agent Web Census

AI crawler blocking on .br websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 103 are .br domains and 63 (n) have a readable robots.txt. 19.0% of those 63 block at least one AI crawler by name across the whole site.

Note: small sample (n=63), treat with caution.

AI crawlers blocked by .br sites with a readable robots.txt (n=63)CCBot14.3%omgili12.7%PetalBot12.7%GPTBot11.1%Google-Extended11.1%Bytespider11.1%cohere-ai11.1%ClaudeBot9.5%Meta-ExternalAgent9.5%Diffbot9.5%anthropic-ai7.9%Claude-Web7.9%Applebot-Extended7.9%Amazonbot7.9%PerplexityBot6.3%

Percentages are of n = 63 .br sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.