Citable

nytimes.com

https://www.nytimes.com/

3 things to fix, worth up to 38 pointsChecked 2026-10-29 15:15 UTCCheck again

Answer engines2 of 7 allowed

  • OAI-SearchBotChatGPT searchblocked
  • ChatGPT-UserChatGPT, when a user asksblocked
  • Claude-SearchBotClaude searchblocked
  • Claude-UserClaude, when a user asksblocked
  • PerplexityBotPerplexityblocked
  • GooglebotGoogle Search, AI Overviews, AI Modeallowed
  • BingbotBing, Copilotallowed

Training crawlers: 0 allowed, 6 blocked. Not scored: it does not change whether answers cite you.

Partial report: the page answered 403, so its contents were not checked.

Fix first · 3

AI answer engines can crawl the page+187 / 25

Fix: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot are blocked, so those engines cannot read this page to cite it.

How to fixIn robots.txt, add a group that allows them, for example: User-agent: OAI-SearchBot Allow: / Training crawlers (GPTBot, ClaudeBot, Google-Extended) are separate tokens: you can keep those blocked.
Details
  • blocked: OAI-SearchBot (ChatGPT search), Disallow: /
  • blocked: ChatGPT-User (ChatGPT, when a user asks), Disallow: /
  • blocked: Claude-SearchBot (Claude search), Disallow: /
  • blocked: Claude-User (Claude, when a user asks), Disallow: /
  • blocked: PerplexityBot (Perplexity), Disallow: /
  • allowed: Googlebot (Google Search, AI Overviews, AI Mode)
  • allowed: Bingbot (Bing, Copilot)

The page can be indexed and quoted+150 / 15

Fix: The site refused Citable's crawler with 403. Bot protection that refuses unknown crawlers often refuses AI crawlers too, and they cannot cite a page they cannot read.

How to fixIn your firewall or bot protection (Cloudflare, Vercel, Akamai and others have a setting for it), let verified AI search crawlers through: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot. Then check again.
Details
  • status 403
  • meta robots: none
  • X-Robots-Tag: none
  • HTTPS

llms.txt+50 / 5

Improve: No /llms.txt.

How to fixAdd /llms.txt: a Markdown file with "# Site name", a "> one-line summary", and lists of links to the pages that matter. https://llmstxt.org
Details
  • llms.txt is a young convention: AI coding tools read it, while no major answer engine says it uses it yet. Cheap to add.

Fix it with your AI

Citable never calls a model. Hand the list to yours, with the exact fixes.

Ask ChatGPTAsk Claude
The text it hands over
Citable checked https://www.nytimes.com/ for AI search (score 17/100). Help me fix these, most important first, with the exact code or config to change:
1. AI answer engines can crawl the page: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot are blocked, so those engines cannot read this page to cite it.
   Suggested fix: In robots.txt, add a group that allows them, for example:
User-agent: OAI-SearchBot
Allow: /
Training crawlers (GPTBot, ClaudeBot, Google-Extended) are separate tokens: you can keep those blocked.
2. The page can be indexed and quoted: The site refused Citable's crawler with 403. Bot protection that refuses unknown crawlers often refuses AI crawlers too, and they cannot cite a page they cannot read.
   Suggested fix: In your firewall or bot protection (Cloudflare, Vercel, Akamai and others have a setting for it), let verified AI search crawlers through: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot. Then check again.
3. llms.txt: No /llms.txt.
   Suggested fix: Add /llms.txt: a Markdown file with "# Site name", a "> one-line summary", and lists of links to the pages that matter. https://llmstxt.org

Passing · 2

  • Sitemap

    Pass: Listed in robots.txt: https://www.nytimes.com/sitemaps/new/news.xml.gz

    5/5
  • Response time

    Pass: The first byte of the page arrived after 293 ms (measured from Frankfurt, redirects included).

    5/5

Notes · 6

  • Readable without JavaScript

    Note: Not checked: the page answered 403.

    0/15
  • Title and description

    Note: Not checked: the page answered 403.

    0/10
  • Structured data (schema.org)

    Note: Not checked: the page answered 403.

    0/10
  • Headings

    Note: Not checked: the page answered 403.

    0/5
  • Canonical address

    Note: Not checked: the page answered 403.

    0/5
  • Model training crawlers

    Note: 6 of 6 training crawlers are blocked. That is a choice, not a score: it does not change whether answers cite you.

    Details
    • blocked: GPTBot (OpenAI model training)
    • blocked: ClaudeBot (Anthropic model training)
    • blocked: Google-Extended (Gemini model training)
    • blocked: Applebot-Extended (Apple model training)
    • blocked: CCBot (Common Crawl)
    • blocked: meta-externalagent (Meta AI)

Show the score

Click the code to select it

AI search score 17 
[![AI search: 17/100](https://citable.flisk.si/badge/nytimes.com.svg)](https://citable.flisk.si/nytimes.com)