Major AI crawlers are blocked in robots.txt

Your robots.txt forbids access to 5 central AI crawlers: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended. Those systems then cannot load your content or use it as a source.

This is what you should do: Check per bot whether the block is intended. For AI visibility, at least GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended should be allowed.

User-agent: GPTBot
Allow: /

User-agent: PerplexityBot
Allow: /

Server blocks bots although robots.txt allows them

For 1 bot(s) from Amazon (Alexa/Rufus), robots.txt says "allowed", but the server responds with 403 or a bot-protection page. Usually a WAF, firewall or CDN rule that locks the crawler out past robots.txt.

This is what you should do: Add the user agents and/or IP ranges of the desired AI crawlers to an allow list in your WAF / Cloudflare / firewall. Verify them via reverse DNS lookup, not just the user-agent string.

Page title too short or too long

The <title> is 66 characters long. Around 30–60 characters is sensible: descriptive enough to name the topic, short enough not to be cut off.

This is what you should do: Write a concise title that contains the topic and, if relevant, the brand.

No llms.txt

There is no file at /llms.txt. llms.txt is a short Markdown overview of your key content that some AI systems use for orientation.

This is what you should do: Create an /llms.txt: an H1 with the name, a short paragraph about the offering, and a link list to the central pages.

# Your company

> One-sentence description.

## Key pages
- [Product](https://your-domain.com/product): …
- [Pricing](https://your-domain.com/pricing): …
- [About](https://your-domain.com/about): …

No Organization schema

There is JSON-LD, but no "Organization" entry. AI systems can then place your brand less well as a distinct entity and link it to external sources.

This is what you should do: Add an "Organization" schema with name, url, logo and sameAs (links to your official profiles).

No FAQ or HowTo markup

Question-and-answer and step-by-step content is harder for AI systems to recognise as directly citable answers without FAQPage or HowTo markup.

This is what you should do: Where the page answers real questions or gives instructions, mark it up as FAQPage or HowTo (no markup without visibly matching content).

No author or authorship signal

No author is recognisable (neither meta tag, rel=author nor JSON-LD Person). For "experience" and "expertise" in the E-E-A-T sense, named, traceable authorship matters.

This is what you should do: For editorial content, name an author and link an author page; also mark them up as JSON-LD "Person".

No visible or marked-up date

No publication or modification date is recognisable (neither <time>, article:published_time nor datePublished in the JSON-LD). For many questions, AI systems prefer current sources.

This is what you should do: For content with a time reference, show a date visibly and mark up datePublished / dateModified in the JSON-LD.