Agent Web Census

Does trustpilot.com block AI crawlers?

Yes — as of 2026-11-18, trustpilot.com's robots.txt blocks 9 of the 27 AI crawlers we track site-wide. It also has a blanket rule blocking all other crawlers.

#338Tranco rank
9of 27 AI crawlers blocked site-wide
11partially restricted or explicitly allowed
nollms.txt present (format check)

Status of each crawler

CrawlerOperatorPurposeStatus in robots.txt
GPTBotOpenAItrainingPartially restricted
ChatGPT-UserOpenAIuser-triggeredPartially restricted
OAI-SearchBotOpenAIsearchPartially restricted
ClaudeBotAnthropictrainingPartially restricted
Claude-UserAnthropicuser-triggeredPartially restricted
Claude-SearchBotAnthropicsearchPartially restricted
anthropic-aiAnthropictrainingBlocked site-wide
Claude-WebAnthropicuser-triggeredBlocked via blanket * rule
Google-ExtendedGoogletrainingPartially restricted
GooglebotGooglesearch (comparison, not AI)Partially restricted
Applebot-ExtendedAppletrainingBlocked site-wide
PerplexityBotPerplexitysearchPartially restricted
Perplexity-UserPerplexityuser-triggeredBlocked via blanket * rule
CCBotCommon CrawltrainingPartially restricted
BytespiderByteDancetrainingBlocked site-wide
AmazonbotAmazontrainingBlocked via blanket * rule
Meta-ExternalAgentMetatrainingPartially restricted
FacebookBotMetatrainingPartially restricted
cohere-aiCohereuser-triggeredBlocked site-wide
DuckAssistBotDuckDuckGouser-triggeredBlocked via blanket * rule
MistralAI-UserMistraluser-triggeredBlocked via blanket * rule
YouBotYou.comsearchBlocked site-wide
DiffbotDiffbottrainingBlocked site-wide
TimpibotTimpitrainingBlocked site-wide
ImagesiftBotHive/ImagesifttrainingBlocked site-wide
omgiliWebz.iotrainingBlocked site-wide
AI2BotAllen Institute for AItrainingBlocked via blanket * rule
PetalBotHuaweisearchBlocked via blanket * rule

“Not named” means no group in the robots.txt names the crawler, so the User-agent: * rules apply (here the * group blocks the whole site).

Other signals

Relevant robots.txt lines

User-agent: FacebookBot
Disallow: /error
Disallow: /evaluate/
Disallow: /evaluate-link
Disallow: /evaluate-unique-link
Disallow: /pingdom
Allow: /reviews/
Disallow: /review/*/transparency
Disallow: /product-reviews/
Disallow: /*?*editmode=
Disallow: /api/*
Disallow: /*?*languages=
Disallow: /*?*stars=
Disallow: /*?*sort=
Disallow: /*?*verified=
Disallow: /*?*topics=
Disallow: /*?*replies=
Disallow: /*?*date=
Disallow: /*?*trustscore=
Disallow: /*?*claimed=
Disallow: /*?*country=
Disallow: /*?*location=
Disallow: /*?*subcategories=
Disallow: /search
Disallow: /sessions
Disallow: /submitted/review
Disallow: /users/
Allow: /users/connect
Disallow: /users/connect?redirect
Disallow: /users/connect?signup
Disallow: /users/connect?intent
Disallow: /users/activate
Disallow: /users/embed-login-to-edit
Disallow: /users/create
Disallow: /unsubscribe/invitation

User-agent: ChatGPT-User
Disallow: /error
Disallow: /evaluate/
Disallow: /evaluate-link
# ... (truncated)

Only groups naming a tracked AI crawler, at most 40 lines, from the scan of 2026-11-18.

Note: robots.txt is a voluntary standard (RFC 9309); it does not technically stop any crawler, and a site may also block bots by other means. The file may have changed since the scan on 2026-11-18; check the live file above. See the methodology for how statuses are derived.

Cite this: Agent Web Census, “Does trustpilot.com block AI crawlers?”, https://agents.datacensus.org/ai-crawlers/site/trustpilot.com/, robots.txt scan of 2026-11-18 (Tranco list V38GN).

← #337 php.net · #341 etsy.com → · all sites · AI crawler statistics

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.