Agent Web Census

How many websites block Googlebot? (Nov 2026)

0.2%of sites with a readable robots.txt block it by name, site-wide
2.9%effectively blocked (incl. * rule)
6.5%partially restricted
2.7%explicitly allowed
#28/28most-blocked rank

Googlebot is operated by Google; purpose: search. Scan of the top 10,000 websites (Tranco list V38GN) on 2026-11-18; repeated weekly.

Key facts

Cite this: Agent Web Census, “How many websites block Googlebot?”, https://agents.datacensus.org/ai-crawlers/googlebot/, scan of the top 10,000 websites (Tranco list V38GN) on 2026-11-18.
Definitions

By popularity rank

Bucketn (readable robots.txt)Fully blockedPartially
top100521.9%19.2%
top1k5140.6%12.5%
top10k5,8060.2%6.5%

By TLD

TLDn (readable robots.txt)Fully blockedNote
.com2,9850.2%
.net2590.8%
.org3100.0%
.ru1520.0%
.io1030.0%
.jp1140.9%
.de1350.0%
.edu1030.0%
.gov800.0%small sample (n=80), treat with caution
.in390.0%small sample (n=39), treat with caution
.cn320.0%small sample (n=32), treat with caution
.br630.0%small sample (n=63), treat with caution
.fr750.0%small sample (n=75), treat with caution
.uk600.0%small sample (n=60), treat with caution
.it581.7%small sample (n=58), treat with caution
.tv470.0%small sample (n=47), treat with caution
.co320.0%small sample (n=32), treat with caution
.es430.0%small sample (n=43), treat with caution
.pl420.0%small sample (n=42), treat with caution
.ai380.0%small sample (n=38), treat with caution

n = domains of that TLD with a readable robots.txt; percentages are of n. TLDs with n < 30 are omitted.

How to block or allow Googlebot

# block
User-agent: Googlebot
Disallow: /

# allow
User-agent: Googlebot
Allow: /

robots.txt is a voluntary standard (RFC 9309); compliance depends on the crawler operator.

← all AI crawlers

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.