geolintv0.4.0AI-search readiness audit
https://cloudflare.com/ → https://www.cloudflare.com/
HTTP 200text/htmlTTFB 462mstotal 1.3srobots.txt 200llms.txt 2002026-09-17 12:55 UTC
Grade A
0 errors·1 warning·10 hints
42/52 checks passed

Categories

AI Crawler Access
96
2 hints
llms.txt
98
1 hint
Structured Data
98
1 hint
Citability
84
1 warning · 5 hints
Technical Foundation
98
1 hint

AI crawler access 51/51 allowed

BotPurposerobots.txtAccess
OpenAI
GPTBottraininghonored✓ allowed
OAI-SearchBotsearchhonored✓ allowed
ChatGPT-Useruser-fetchmay bypass✓ allowed
OAI-AdsBotmixedhonored✓ allowed
Anthropic
ClaudeBottraininghonored✓ allowed
Claude-Useruser-fetchhonored✓ allowed
Claude-SearchBotsearchhonored✓ allowed
anthropic-airetiredtrainingunverified✓ allowed
Claude-Webretiredsearchunverified✓ allowed
Perplexity
PerplexityBotsearchhonored✓ allowed
Perplexity-Useruser-fetchmay bypass✓ allowed
Google
Googlebotsearchhonored✓ allowed
Google-Extendedopt-outtraininghonored✓ allowed
Apple
Applebotsearchhonored✓ allowed
Applebot-Extendedopt-outtraininghonored✓ allowed
Meta
Meta-ExternalAgentmeta-externalagenttraininghonored✓ allowed
Meta-WebIndexersearchhonored✓ allowed
Meta-ExternalFetcheruser-fetchmay bypass✓ allowed
facebookexternalhituser-fetchunverified✓ allowed
FacebookBotretiredtrainingunverified✓ allowed
Amazon
Amazonbotmixedhonored✓ allowed
Amzn-SearchBotsearchhonored✓ allowed
Amzn-Useruser-fetchhonored✓ allowed
Mistral AI
MistralAI-Useruser-fetchunverified✓ allowed
MistralAI-Indexsearchunverified✓ allowed
MistralAI-Trainingtrainingunverified✓ allowed
Microsoft
Bingbotsearchhonored✓ allowed
DuckDuckGo
DuckDuckBotsearchhonored✓ allowed
DuckAssistBotuser-fetchhonored✓ allowed
You.com
YouBotsearchhonored✓ allowed
Moonshot AI
KimiBottrainingunverified✓ allowed
Kimi-Useruser-fetchunverified✓ allowed
Alibaba
TongyiBotmixedunverified✓ allowed
Exa
ExaBotsearchunverified✓ allowed
Tavily
TavilyBotsearchunverified✓ allowed
Timpi
Timpibotsearchunverified✓ allowed
ByteDance
Bytespidermixedunverified✓ allowed
Common Crawl
CCBottraininghonored✓ allowed
Cohere
cohere-aiuser-fetchunverified✓ allowed
cohere-training-data-crawlertrainingunverified✓ allowed
Diffbot
Diffbotmixedhonored✓ allowed
Webz.io
Omgilibotomgilibotretiredtraininghonored✓ allowed
webziotraininghonored✓ allowed
Webzio-Extendedopt-outtraininghonored✓ allowed
The Hive
ImagesiftBottraininghonored✓ allowed
Huawei
PanguBottrainingunverified✓ allowed
Kangaroo LLM
Kangaroo Bottrainingunverified✓ allowed
Allen Institute for AI
AI2Bottraininghonored✓ allowed
Ai2Bot-Dolmatraininghonored✓ allowed
NICT
ICC-Crawlertrainingunverified✓ allowed
Velen
VelenPublicWebCrawlertrainingunverified✓ allowed

Findings

⚠Warnings 1
⚠ warningcontent/missing-datesCitability
Visible publication/modified date

No machine-readable publication or modified date found

Checked <time datetime>, article:* meta tags and JSON-LD datePublished/dateModified — none present.

FixExpose a date via <time datetime="…">, article:published_time meta or JSON-LD datePublished/dateModified.

ℹHints 10
ℹ hintai-crawler/stale-tokensAI Crawler Access
Retired AI bot tokens in robots.txt

Stale token 'anthropic-ai' — Anthropic now uses 'ClaudeBot'

'anthropic-ai' matched retired token anthropic-ai; this rule has no effect on current Anthropic crawlers.

FixReplace 'User-agent: anthropic-ai' with 'User-agent: ClaudeBot' (or delete the stale block).

User-agent: anthropic-ai
ℹ hintai-crawler/stale-tokensAI Crawler Access
Retired AI bot tokens in robots.txt

Stale token 'claude-web' — Anthropic now uses 'Claude-SearchBot'

'claude-web' matched retired token Claude-Web; this rule has no effect on current Anthropic crawlers.

FixReplace 'User-agent: claude-web' with 'User-agent: Claude-SearchBot' (or delete the stale block).

User-agent: claude-web
ℹ hintllms-txt/ai-manifestllms.txt
Emerging AI manifest files (agents.json, ai.txt)

Emerging AI manifest file detected: /agents.json

These conventions have no settled spec yet — they neither help nor hurt citations today, but their contents should be intentional rather than forgotten leftovers.

FixVerify the manifest contents are accurate and current; /llms.txt remains the broadly recognized file to invest in.

/agents.json
ℹ hintschema/no-breadcrumbStructured Data
BreadcrumbList schema present

No BreadcrumbList schema found

Breadcrumbs expose site structure to machines; without them a page floats without context.

FixAdd BreadcrumbList JSON-LD matching the visible breadcrumb trail.

ℹ hintcontent/no-h1Citability
Single descriptive H1

2 <h1> elements on the page

A single H1 anchors the page topic; several competing H1s dilute the signal machines use to understand it.

FixKeep one descriptive <h1>; demote the others to <h2> or below.

ℹ hintcontent/no-question-headingsCitability
Question-shaped headings

No question-shaped headings (h2/h3 ending in "?")

Question headings map directly onto user prompts and make sections self-contained citable answers.

FixRewrite a few section headings as the actual questions your audience asks.

ℹ hintcontent/no-authorCitability
Author attribution

No author attribution found

No meta[name=author], rel=author, JSON-LD author or author/byline markup — the content looks anonymous.

FixAttribute the content: an author byline, meta[name=author] or a Person in JSON-LD author.

ℹ hintcontent/no-structureCitability
Structured content (lists/tables)

Long page with no lists or tables

Bulleted lists and tables give answer engines pre-structured, quotable units of content.

FixConvert enumerations and comparisons into <ul>/<ol> lists or <table> markup.

ℹ hintcontent/answer-firstCitability
Page leads with a direct answer

Lead paragraph after the H1 is thin (~15 words)

A teaser or caption-length opener is hard to quote verbatim; engines prefer a self-contained answer they can lift whole.

FixExpand the opening paragraph into a direct, self-contained answer (roughly 40–80 words) before diving into details.

One platform for your apps, agents, and workforce. Build, secure, and scale without managing infrastructure.
ℹ hinttechnical/redirectTechnical Foundation
No unexpected redirect

The URL redirected to a different final URL

Requested https://cloudflare.com/, ended at https://www.cloudflare.com/.

FixMake sure redirects are intentional, single-hop and point at the canonical destination.

https://cloudflare.com/ → https://www.cloudflare.com/