Agent Web Census

AI crawler blocking on .pl websites (Nov 2026)

Among the top 10,000 websites (scan of 2026-11-18), 54 are .pl domains and 42 (n) have a readable robots.txt. 38.1% of those 42 block at least one AI crawler by name across the whole site.

Note: small sample (n=42), treat with caution.

AI crawlers blocked by .pl sites with a readable robots.txt (n=42)GPTBot26.2%CCBot19.0%Bytespider14.3%PetalBot11.9%ClaudeBot9.5%Amazonbot9.5%Meta-ExternalAgent9.5%cohere-ai9.5%YouBot9.5%Diffbot9.5%Timpibot9.5%Google-Extended7.1%DuckAssistBot7.1%ImagesiftBot7.1%omgili7.1%

Percentages are of n = 42 .pl sites with a readable robots.txt. Small samples are noisy.

Definitions

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.