technical/http-errorTechnical FoundationPage returned HTTP 403
Crawlers receive the error response, not your content.
FixReturn a 200 with real content; fix the underlying 4xx/5xx or the rules that trigger it.
https://openai.com/ → HTTP 403
| Bot | Purpose | robots.txt | Access |
|---|---|---|---|
| OpenAI | |||
GPTBot | training | honored | ✓ allowed |
OAI-SearchBot | search | honored | ✓ allowed |
ChatGPT-User | user-fetch | may bypass | ✓ allowed |
OAI-AdsBot | mixed | honored | ✓ allowed |
| Anthropic | |||
ClaudeBot | training | honored | ✓ allowed |
Claude-User | user-fetch | honored | ✓ allowed |
Claude-SearchBot | search | honored | ✓ allowed |
anthropic-airetired | training | unverified | ✓ allowed |
Claude-Webretired | search | unverified | ✓ allowed |
| Perplexity | |||
PerplexityBot | search | honored | ✓ allowed |
Perplexity-User | user-fetch | may bypass | ✓ allowed |
Googlebot | search | honored | ✓ allowed |
Google-Extendedopt-out | training | honored | ✓ allowed |
| Apple | |||
Applebot | search | honored | ✓ allowed |
Applebot-Extendedopt-out | training | honored | ✓ allowed |
| Meta | |||
Meta-ExternalAgentmeta-externalagent | training | honored | ✓ allowed |
Meta-WebIndexer | search | honored | ✓ allowed |
Meta-ExternalFetcher | user-fetch | may bypass | ✓ allowed |
facebookexternalhit | user-fetch | unverified | ✓ allowed |
FacebookBotretired | training | unverified | ✓ allowed |
| Amazon | |||
Amazonbot | mixed | honored | ✓ allowed |
Amzn-SearchBot | search | honored | ✓ allowed |
Amzn-User | user-fetch | honored | ✓ allowed |
| Mistral AI | |||
MistralAI-User | user-fetch | unverified | ✓ allowed |
MistralAI-Index | search | unverified | ✓ allowed |
MistralAI-Training | training | unverified | ✓ allowed |
| Microsoft | |||
Bingbot | search | honored | ✓ allowed |
| DuckDuckGo | |||
DuckDuckBot | search | honored | ✓ allowed |
DuckAssistBot | user-fetch | honored | ✓ allowed |
| You.com | |||
YouBot | search | honored | ✓ allowed |
| Moonshot AI | |||
KimiBot | training | unverified | ✓ allowed |
Kimi-User | user-fetch | unverified | ✓ allowed |
| Alibaba | |||
TongyiBot | mixed | unverified | ✓ allowed |
| Exa | |||
ExaBot | search | unverified | ✓ allowed |
| Tavily | |||
TavilyBot | search | unverified | ✓ allowed |
| Timpi | |||
Timpibot | search | unverified | ✓ allowed |
| ByteDance | |||
Bytespider | mixed | unverified | ✓ allowed |
| Common Crawl | |||
CCBot | training | honored | ✓ allowed |
| Cohere | |||
cohere-ai | user-fetch | unverified | ✓ allowed |
cohere-training-data-crawler | training | unverified | ✓ allowed |
| Diffbot | |||
Diffbot | mixed | honored | ✓ allowed |
| Webz.io | |||
Omgilibotomgilibotretired | training | honored | ✓ allowed |
webzio | training | honored | ✓ allowed |
Webzio-Extendedopt-out | training | honored | ✓ allowed |
| The Hive | |||
ImagesiftBot | training | honored | ✓ allowed |
| Huawei | |||
PanguBot | training | unverified | ✓ allowed |
| Kangaroo LLM | |||
Kangaroo Bot | training | unverified | ✓ allowed |
| Allen Institute for AI | |||
AI2Bot | training | honored | ✓ allowed |
Ai2Bot-Dolma | training | honored | ✓ allowed |
| NICT | |||
ICC-Crawler | training | unverified | ✓ allowed |
| Velen | |||
VelenPublicWebCrawler | training | unverified | ✓ allowed |
technical/http-errorTechnical FoundationPage returned HTTP 403
Crawlers receive the error response, not your content.
FixReturn a 200 with real content; fix the underlying 4xx/5xx or the rules that trigger it.
https://openai.com/ → HTTP 403
technical/client-renderedTechnical FoundationPage appears client-rendered — AI crawlers see an empty shell
Only ~0 words of visible text in the initial HTML — likely rendered client-side.
FixUse server-side rendering, static generation or prerendering so the real content is in the first HTML response.
0 words of visible text
llms-txt/missingllms.txtNo llms.txt found
GET https://openai.com/llms.txt returned HTTP 403.
FixCreate /llms.txt at the site root: an H1 title, a short blockquote summary, and ## sections linking to your key content.
https://openai.com/llms.txt → HTTP 403
schema/no-jsonldStructured DataNo JSON-LD structured data on the page
AI engines use schema.org markup to identify entities, dates, authors and page types. Without it they fall back to unreliable inference.
FixAdd JSON-LD blocks describing the page (Article, Organization, BreadcrumbList, FAQPage as appropriate).
content/thin-contentCitabilityThin content — only ~0 words of visible text
Pages with very little extractable text give AI engines nothing to quote; they also correlate with client-rendered shells.
FixExpand the page with substantive, answerable content — explanations, specifics, Q&A.
0 words
content/no-h1CitabilityNo <h1> on the page
The H1 anchors how machines understand and index the page topic.
FixAdd one descriptive <h1> that states what the page is about.
content/missing-datesCitabilityNo machine-readable publication or modified date found
Checked <time datetime>, article:* meta tags and JSON-LD datePublished/dateModified — none present.
FixExpose a date via <time datetime="…">, article:published_time meta or JSON-LD datePublished/dateModified.
technical/title-missingTechnical FoundationMissing or empty <title>
The title is the primary human-readable label attached to the page in AI answers.
FixAdd a descriptive <title> that names the page and its purpose.
llms-txt/llms-full-missingllms.txtNo llms-full.txt found
GET https://openai.com/llms-full.txt returned HTTP 403. The optional full-content companion is absent.
FixConsider publishing /llms-full.txt with your key documentation inlined as markdown.
https://openai.com/llms-full.txt → HTTP 403
llms-txt/ai-manifestllms.txtNo emerging AI manifest files found
Checked /agents.json, /ai.txt, /.well-known/ai-plugin.json — all absent. These conventions are early and entirely optional.
FixPrioritize /llms.txt; add agents.json or ai.txt only if they fit your AI distribution strategy.
schema/no-organizationStructured DataNo Organization schema found
Without entity grounding, engines cannot connect this site to a known organization — weaker trust signals for citation.
FixAdd an Organization JSON-LD block (name, url, logo, sameAs links) at least on the homepage.
schema/no-breadcrumbStructured DataNo BreadcrumbList schema found
Breadcrumbs expose site structure to machines; without them a page floats without context.
FixAdd BreadcrumbList JSON-LD matching the visible breadcrumb trail.
content/no-question-headingsCitabilityNo question-shaped headings (h2/h3 ending in "?")
Question headings map directly onto user prompts and make sections self-contained citable answers.
FixRewrite a few section headings as the actual questions your audience asks.
content/no-authorCitabilityNo author attribution found
No meta[name=author], rel=author, JSON-LD author or author/byline markup — the content looks anonymous.
FixAttribute the content: an author byline, meta[name=author] or a Person in JSON-LD author.
content/no-data-pointsCitabilityFew concrete data points found (0 numeric stats)
Percentages, growth figures and quantities give AI engines precise, attributable claims to quote.
FixAdd verifiable numbers — percentages, counts, benchmarks — near the claims they support.
content/lang-missingCitabilityThe <html> element has no lang attribute
Without a declared language, engines must infer it — and can get it wrong.
FixAdd lang to the root element, e.g. <html lang="en">.
technical/canonicalTechnical FoundationNo canonical link element
Without a canonical, engines pick one themselves — and may consolidate onto the wrong URL.
FixAdd <link rel="canonical" href="…"> pointing at the preferred URL for this page.
technical/meta-descriptionTechnical FoundationNo meta description
A 1–2 sentence description gives answer engines a clean, pre-written summary of the page.
FixWrite a meta[name=description] of ~50–160 characters summarizing the page.