Checked 9/24/2026 · https://www.nytimes.com/
There are 2 critical blockers for AI discoverability — fix these first.
AI crawler matrix
For each bot: what your robots.txt allows — and what your server actually returns to the bot's user agent. Rows highlighted in red: robots allows it, but the server blocks it (usually a WAF or bot-protection rule).
| Bot | Operator | robots.txt | Server response |
|---|---|---|---|
| GPTBot | OpenAI | blocked | blocked (403 / WAF) |
| OAI-SearchBot | OpenAI | blocked | blocked (403 / WAF) |
| ChatGPT-User | OpenAI | blocked | blocked (403 / WAF) |
| ClaudeBot | Anthropic | blocked | blocked (403 / WAF) |
| Claude-SearchBot | Anthropic | blocked | blocked (403 / WAF) |
| Claude-User | Anthropic | blocked | blocked (403 / WAF) |
| PerplexityBot | Perplexity | blocked | blocked (403 / WAF) |
| Perplexity-User | Perplexity | blocked | blocked (403 / WAF) |
| Google-Extended | Google (Gemini-Training) | blocked | — |
| Googlebot | Google (KI-Übersichten) | allowed | blocked (403 / WAF) |
| CCBot | Common Crawl (Trainingsdaten vieler LLMs) | blocked | blocked (403 / WAF) |
| Bytespider | ByteDance (Doubao) | blocked | blocked (403 / WAF) |
| Amazonbot | Amazon (Alexa/Rufus) | allowed | blocked (403 / WAF) |
| Applebot-Extended | Apple (Intelligence-Training) | blocked | — |
| Meta-ExternalAgent | Meta (Llama/Meta AI) | blocked | blocked (403 / WAF) |
How is the score calculated?
| AI search bots are blocked in robots.txt | -32 |
| Server blocks bots although robots.txt allows them | -12 |
| Very little text in the HTML | -10 |
| No structured data (JSON-LD) | -8 |
| Page title too short or too long | -6 |
| Not exactly one H1 heading | -5 |
| Meta description missing or unsuitable | -5 |
| No llms.txt | -5 |
| No author or authorship signal | -4 |
| No sameAs links | -3 |
| No link to legal notice / about / contact | -3 |
| No visible or marked-up date | -3 |
| No canonical URL | -3 |
| Result | 1 / 100 |
Access for AI crawlers
AI search bots are blocked in robots.txt
Your robots.txt blocks 6 bots that fetch content for answers in AI assistants and search engines: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User. While they are blocked, your site cannot appear there as a source.
This is what you should do: Check for each bot whether the block is intended. If you only want to opt out of AI training, block the training bots (GPTBot, ClaudeBot, Google-Extended) and allow the search bots.
User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: /
Server blocks bots although robots.txt allows them
For 2 bot(s) from Google (KI-Übersichten), Amazon (Alexa/Rufus), robots.txt says "allowed", but the server responds with 403 or a bot-protection page. Usually a WAF, firewall or CDN rule that locks the crawler out past robots.txt.
This is what you should do: Add the user agents and/or IP ranges of the desired AI crawlers to an allow list in your WAF / Cloudflare / firewall. Verify them via reverse DNS lookup, not just the user-agent string.
AI training opted out
Your robots.txt blocks crawlers that collect content for training language models: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended. This does not prevent citations — search and retrieval bots such as OAI-SearchBot, PerplexityBot and Googlebot for AI Overviews are separate.
This is what you should do: Nothing to do if the opt-out is intended. Just make sure the search bots stay allowed.
Content without JavaScript
Very little text in the HTML
The initial HTML contains only around 55 characters of visible text. That's barely enough for AI systems to grasp the page's topic with confidence.
This is what you should do: Make sure the actual content is fully in the HTML and not in images, iframes or blocks loaded via JavaScript.
Structure & machine understanding
No structured data (JSON-LD)
The page contains no JSON-LD. Structured data helps AI systems recognise entities, author, organisation, date and page type unambiguously — without it, everything has to be guessed from the body text.
This is what you should do: Add at least an "Organization" schema and one that fits the page type (Article, Product, FAQPage …) as JSON-LD in the <head>.
<script type="application/ld+json">
{"@context":"https://schema.org","@type":"Organization",
"name":"Your company","url":"https://your-domain.com",
"sameAs":["https://www.linkedin.com/company/…"]}
</script>Page title too short or too long
The <title> is 11 characters long. Around 30–60 characters is sensible: descriptive enough to name the topic, short enough not to be cut off.
This is what you should do: Write a concise title that contains the topic and, if relevant, the brand.
Not exactly one H1 heading
The page has 0 H1 headings. A single, unambiguous H1 tells humans and machines what the page is primarily about.
This is what you should do: Set exactly one H1 that names the page's main topic. Everything below starts at H2.
Meta description missing or unsuitable
The meta description is 0 characters long. It is often the first thing AI systems and search engines read as a summary of the page.
This is what you should do: Write a standalone summary of the page in one or two sentences (around 120–200 characters), not a string of keywords.
No llms.txt
There is no file at /llms.txt. llms.txt is a short Markdown overview of your key content that some AI systems use for orientation.
This is what you should do: Create an /llms.txt: an H1 with the name, a short paragraph about the offering, and a link list to the central pages.
# Your company > One-sentence description. ## Key pages - [Product](https://your-domain.com/product): … - [Pricing](https://your-domain.com/pricing): … - [About](https://your-domain.com/about): …
Trust and entity signals (E-E-A-T)
No author or authorship signal
No author is recognisable (neither meta tag, rel=author nor JSON-LD Person). For "experience" and "expertise" in the E-E-A-T sense, named, traceable authorship matters.
This is what you should do: For editorial content, name an author and link an author page; also mark them up as JSON-LD "Person".
No sameAs links
The JSON-LD is missing "sameAs". It lets AI systems clearly map your brand or person to known entities (Wikidata, LinkedIn, industry directories).
This is what you should do: Add a "sameAs" in Organization or Person with the URLs of your official profiles and directory entries.
No link to legal notice / about / contact
No link to a legal notice, "About" or contact page is recognisable on the checked page. Such pages are a strong trust signal and help with entity mapping.
This is what you should do: Link the legal notice or "About" and contact clearly, usually in the footer of every page.
No visible or marked-up date
No publication or modification date is recognisable (neither <time>, article:published_time nor datePublished in the JSON-LD). For many questions, AI systems prefer current sources.
This is what you should do: For content with a time reference, show a date visibly and mark up datePublished / dateModified in the JSON-LD.
Technical foundation
No canonical URL
The page sets no <link rel="canonical">. Without a canonical, it can stay unclear which version is authoritative when several URL variants exist.
This is what you should do: Set a self-referencing <link rel="canonical"> with the preferred URL on every page.
OK (3)
- The sitemap is reachable.
- Fast server response time.
- No noindex — the page may be indexed.
Next steps
- AI search bots are blocked in robots.txt. Check for each bot whether the block is intended. If you only want to opt out of AI training, block the training bots (GPTBot, ClaudeBot, Google-Extended) and allow the search bots.
- Server blocks bots although robots.txt allows them. Add the user agents and/or IP ranges of the desired AI crawlers to an allow list in your WAF / Cloudflare / firewall. Verify them via reverse DNS lookup, not just the user-agent string.
- Very little text in the HTML. Make sure the actual content is fully in the HTML and not in images, iframes or blocks loaded via JavaScript.
Weekly re-scan with an email alert on every change. In preparation — add your address.
Method & limits
Checked 9/24/2026. citeglass fetches nytimes.com and its related files (robots.txt, llms.txt, sitemap.xml) over HTTP — once as a normal browser, once per AI crawler user agent. No JavaScript is executed. Time to first byte: 99 ms.
What this is not: No rank or citation tracking, no statement about whether a model actually names you, and no check of content loaded via JavaScript. A snapshot from the perspective of one server IP.