Agent Web Census

Does nzherald.co.nz block AI crawlers?

Yes — as of 2026-11-18, nzherald.co.nz's robots.txt blocks 24 of the 27 AI crawlers we track site-wide, including GPTBot and ClaudeBot.

#3396Tranco rank
24of 27 AI crawlers blocked site-wide
0partially restricted or explicitly allowed
nollms.txt present (format check)

Status of each crawler

CrawlerOperatorPurposeStatus in robots.txt
GPTBotOpenAItrainingBlocked site-wide
ChatGPT-UserOpenAIuser-triggeredBlocked site-wide
OAI-SearchBotOpenAIsearchBlocked site-wide
ClaudeBotAnthropictrainingBlocked site-wide
Claude-UserAnthropicuser-triggeredBlocked site-wide
Claude-SearchBotAnthropicsearchBlocked site-wide
anthropic-aiAnthropictrainingBlocked site-wide
Claude-WebAnthropicuser-triggeredBlocked site-wide
Google-ExtendedGoogletrainingBlocked site-wide
GooglebotGooglesearch (comparison, not AI)Partially restricted
Applebot-ExtendedAppletrainingBlocked site-wide
PerplexityBotPerplexitysearchBlocked site-wide
Perplexity-UserPerplexityuser-triggeredBlocked site-wide
CCBotCommon CrawltrainingBlocked site-wide
BytespiderByteDancetrainingBlocked site-wide
AmazonbotAmazontrainingBlocked site-wide
Meta-ExternalAgentMetatrainingBlocked site-wide
FacebookBotMetatrainingNot named (default rules apply)
cohere-aiCohereuser-triggeredBlocked site-wide
DuckAssistBotDuckDuckGouser-triggeredBlocked site-wide
MistralAI-UserMistraluser-triggeredBlocked site-wide
YouBotYou.comsearchBlocked site-wide
DiffbotDiffbottrainingBlocked site-wide
TimpibotTimpitrainingBlocked site-wide
ImagesiftBotHive/ImagesifttrainingNot named (default rules apply)
omgiliWebz.iotrainingBlocked site-wide
AI2BotAllen Institute for AItrainingBlocked site-wide
PetalBotHuaweisearchNot named (default rules apply)

“Not named” means no group in the robots.txt names the crawler, so the User-agent: * rules apply.

Other signals

Relevant robots.txt lines

User-agent: anthropic-ai
Allow: /sponsored-stories/*
Disallow: /fakebots
Disallow: /*keyword=*
Disallow: /*print.cfm*
Disallow: /*NaN/*
Disallow: /nzh-search/
Disallow: */my-account/profile*
Disallow: */my-account/reset-password
Disallow: */my-account/register/newsletters-setup
Disallow: */my-account/register/profile-setup
Disallow: */my-account/subscription/checkout/register/
Disallow: */my-account/subscription/checkout/sign-in/
Disallow: */my-account/subscription/checkout/
Disallow: */my-account/subscription/payment/
Disallow: */my-account/subscription/confirmation/
Disallow: */my-account/subscription/offers/sign-in/
Disallow: */my-account/subscription/offers/?campaignId=*
Disallow: /app/my-account/register
Disallow: /app/my-account/sign-in
Disallow: */premium/*
Disallow: /

User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Claude-Web
Allow: /sponsored-stories/*
Disallow: /fakebots
Disallow: /*keyword=*
Disallow: /*print.cfm*
Disallow: /*NaN/*
Disallow: /nzh-search/
Disallow: */my-account/profile*
Disallow: */my-account/reset-password
Disallow: */my-account/register/newsletters-setup
Disallow: */my-account/register/profile-setup
Disallow: */my-account/subscription/checkout/register/
Disallow: */my-account/subscription/checkout/sign-in/
Disallow: */my-account/subscription/checkout/
# ... (truncated)

Only groups naming a tracked AI crawler, at most 40 lines, from the scan of 2026-11-18.

Note: robots.txt is a voluntary standard (RFC 9309); it does not technically stop any crawler, and a site may also block bots by other means. The file may have changed since the scan on 2026-11-18; check the live file above. See the methodology for how statuses are derived.

Cite this: Agent Web Census, “Does nzherald.co.nz block AI crawlers?”, https://agents.datacensus.org/ai-crawlers/site/nzherald.co.nz/, robots.txt scan of 2026-11-18 (Tranco list V38GN).

← #3395 geeksforgeeks.org · #3398 zdf.de → · all sites · AI crawler statistics

Last updated: (UTC). Numbers are recomputed automatically from primary sources; see methodology.