Checked 9/3/2026 · https://www.iana.org/
No hard blockers, but 2 points weaken AI discoverability.
AI crawler matrix
For each bot: what your robots.txt allows — and what your server actually returns to the bot's user agent. Rows highlighted in red: robots allows it, but the server blocks it (usually a WAF or bot-protection rule).
| Bot | Operator | robots.txt | Server response |
|---|---|---|---|
| GPTBot | OpenAI | allowed (via User-agent: *) | 200 + content |
| OAI-SearchBot | OpenAI | allowed (via User-agent: *) | 200 + content |
| ChatGPT-User | OpenAI | allowed (via User-agent: *) | 200 + content |
| ClaudeBot | Anthropic | allowed (via User-agent: *) | 200 + content |
| Claude-SearchBot | Anthropic | allowed (via User-agent: *) | 200 + content |
| Claude-User | Anthropic | allowed (via User-agent: *) | 200 + content |
| PerplexityBot | Perplexity | allowed (via User-agent: *) | 200 + content |
| Perplexity-User | Perplexity | allowed (via User-agent: *) | 200 + content |
| Google-Extended | Google (Gemini-Training) | allowed (via User-agent: *) | — |
| Googlebot | Google (KI-Übersichten) | allowed (via User-agent: *) | 200 + content |
| CCBot | Common Crawl (Trainingsdaten vieler LLMs) | allowed (via User-agent: *) | 200 + content |
| Bytespider | ByteDance (Doubao) | allowed (via User-agent: *) | 200 + content |
| Amazonbot | Amazon (Alexa/Rufus) | allowed (via User-agent: *) | 200 + content |
| Applebot-Extended | Apple (Intelligence-Training) | allowed (via User-agent: *) | — |
| Meta-ExternalAgent | Meta (Llama/Meta AI) | allowed (via User-agent: *) | 200 + content |
How is the score calculated?
| No structured data (JSON-LD) | -8 |
| Meta description missing or unsuitable | -5 |
| No llms.txt | -5 |
| No author or authorship signal | -4 |
| No sitemap.xml found | -4 |
| No sameAs links | -3 |
| No visible or marked-up date | -3 |
| Result | 68 / 100 |
Structure & machine understanding
No structured data (JSON-LD)
The page contains no JSON-LD. Structured data helps AI systems recognise entities, author, organisation, date and page type unambiguously — without it, everything has to be guessed from the body text.
This is what you should do: Add at least an "Organization" schema and one that fits the page type (Article, Product, FAQPage …) as JSON-LD in the <head>.
<script type="application/ld+json">
{"@context":"https://schema.org","@type":"Organization",
"name":"Your company","url":"https://your-domain.com",
"sameAs":["https://www.linkedin.com/company/…"]}
</script>Meta description missing or unsuitable
The meta description is 0 characters long. It is often the first thing AI systems and search engines read as a summary of the page.
This is what you should do: Write a standalone summary of the page in one or two sentences (around 120–200 characters), not a string of keywords.
No llms.txt
There is no file at /llms.txt. llms.txt is a short Markdown overview of your key content that some AI systems use for orientation.
This is what you should do: Create an /llms.txt: an H1 with the name, a short paragraph about the offering, and a link list to the central pages.
# Your company > One-sentence description. ## Key pages - [Product](https://your-domain.com/product): … - [Pricing](https://your-domain.com/pricing): … - [About](https://your-domain.com/about): …
Trust and entity signals (E-E-A-T)
No author or authorship signal
No author is recognisable (neither meta tag, rel=author nor JSON-LD Person). For "experience" and "expertise" in the E-E-A-T sense, named, traceable authorship matters.
This is what you should do: For editorial content, name an author and link an author page; also mark them up as JSON-LD "Person".
No sameAs links
The JSON-LD is missing "sameAs". It lets AI systems clearly map your brand or person to known entities (Wikidata, LinkedIn, industry directories).
This is what you should do: Add a "sameAs" in Organization or Person with the URLs of your official profiles and directory entries.
No visible or marked-up date
No publication or modification date is recognisable (neither <time>, article:published_time nor datePublished in the JSON-LD). For many questions, AI systems prefer current sources.
This is what you should do: For content with a time reference, show a date visibly and mark up datePublished / dateModified in the JSON-LD.
Technical foundation
No sitemap.xml found
No valid XML sitemap was reachable at /sitemap.xml and robots.txt names none. A sitemap helps crawlers find all relevant URLs.
This is what you should do: Generate a sitemap.xml with all indexable URLs and reference it in robots.txt.
OK (6)
- The main content is in the HTML without JavaScript.
- Exactly one H1 heading.
- A canonical URL is set.
- The page title has a sensible length.
- Fast server response time.
- No noindex — the page may be indexed.
Next steps
- No structured data (JSON-LD). Add at least an "Organization" schema and one that fits the page type (Article, Product, FAQPage …) as JSON-LD in the <head>.
- Meta description missing or unsuitable. Write a standalone summary of the page in one or two sentences (around 120–200 characters), not a string of keywords.
- No llms.txt. Create an /llms.txt: an H1 with the name, a short paragraph about the offering, and a link list to the central pages.
Weekly re-scan with an email alert on every change. In preparation — add your address.
Method & limits
Checked 9/3/2026. citeglass fetches iana.org and its related files (robots.txt, llms.txt, sitemap.xml) over HTTP — once as a normal browser, once per AI crawler user agent. No JavaScript is executed. Time to first byte: 95 ms.
What this is not: No rank or citation tracking, no statement about whether a model actually names you, and no check of content loaded via JavaScript. A snapshot from the perspective of one server IP.