A practical, copy-paste guide to the four infrastructure fixes that determine whether ChatGPT, Perplexity, and Google AI Overviews cite your site or skip it.
I audited 360 domains against an AI-search-readiness framework. The average score was 54.1 out of 100. Most sites fail not because of content quality, but because of four infrastructure issues that take 30 minutes to fix.
This is the quick version. Four steps, real code, in order.
Why 30 Minutes
AI search is a selection problem, not a ranking problem. A model receives a query, decides which sources to consult, extracts a passage, and either cites you or paraphrases without attribution. There is no page two.
Getting selected requires four things to work in sequence:
- Access — the AI crawler can reach your content
- Orientation — it can find what matters
- Understanding — it can parse what you are
- Quotability — it can extract a self-contained passage
Fix access before schema. Fix schema before content. Wrong order = wasted work. I saw teams rewriting content for "AI optimization" while their robots.txt blocked GPTBot.
Let's go.
Step 1: robots.txt (5 minutes)
There are at least 11 AI crawlers actively indexing the web. Each has a distinct user-agent token. Most sites block them by accident — a CMS, a security plugin, or a boilerplate template added Disallow: / and nobody reviewed it.
Check what you're serving right now:
curl https://yoursite.com/robots.txt
If you see Disallow: / under User-agent: *, you're blocking everything. If you see Disallow: / with no user-agent specified, same thing.
Here's a minimal robots.txt that explicitly allows the major AI crawlers while keeping private paths locked down:
# Allow major AI crawlers access to public content
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Applebot
Allow: /
User-agent: CCBot
Allow: /
User-agent: Bytespider
Allow: /
User-agent: Diffbot
Allow: /
# Block private/admin paths from all crawlers
User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/internal/
# Sitemap
Sitemap: https://yoursite.com/sitemap.xml
Upload this to your site root. Done.
The Rendering Trap
One more thing on access: if your site is a SPA that returns an empty <div id="root"></div> on the initial HTML response, AI crawlers see nothing. GPTBot and PerplexityBot do not execute JavaScript reliably.
Check what a crawler actually sees:
curl -A "GPTBot" https://yoursite.com | grep -i "your main heading"
If that returns nothing, your content is invisible. You need SSR, SSG, or a prerendering layer. This is not a 5-minute fix, but you need to know about it now.
Step 2: llms.txt (5 minutes)
llms.txt is a plain-text file at the root of your site that gives AI crawlers a structured summary of your content. Think of it as a sitemap that a model can actually read.
In my audit, 54.2% of sites had a llms.txt file, but only 26.9% had a complete one. A broken llms.txt is worse than none — it sends a model a map with missing streets.
Here's a minimal template:
# Your Company Name
> Your Company builds [what you build] for [who you serve].
## Docs
- [Getting Started](https://yoursite.com/docs/getting-started): Quick start guide
- [API Reference](https://yoursite.com/docs/api): Full REST API documentation
## Product
- [Features](https://yoursite.com/features): Feature overview
- [Pricing](https://yoursite.com/pricing): Pricing tiers and FAQ
## Blog
- [Blog Index](https://yoursite.com/blog): Engineering and product blog
Key rules:
- The
#line is your site title - The
>line is a one-sentence summary of what you do -
##sections group links - Each link is
- [Title](URL): Description— the description matters, it gives the model context about what's at that URL
Don't list every blog post. List the pages that answer "what is this site?" and "what does it do?"
Upload to https://yoursite.com/llms.txt. Done.
Step 3: Schema.org JSON-LD (10 minutes)
Schema is how you tell a model what your entities are: who you are, what your site does, what questions your pages answer.
In the audit, 75.6% of sites had some schema, but only 52.2% had Organization schema and 18.1% had FAQ schema. The gap between "has some schema" and "has the schema types that matter for AI citation" is where most sites lose ground.
Organization + WebSite Schema
Drop this in the <head> of your homepage. Replace the values with your real data.
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@graph": [
{
"@type": "Organization",
"@id": "https://yoursite.com/#organization",
"name": "Your Company",
"url": "https://yoursite.com",
"logo": "https://yoursite.com/logo.png",
"description": "Your Company builds [what you build].",
"sameAs": [
"https://github.com/yourcompany",
"https://x.com/yourcompany",
"https://www.linkedin.com/company/yourcompany"
],
"contactPoint": {
"@type": "ContactPoint",
"contactType": "support",
"email": "support@yoursite.com",
"url": "https://yoursite.com/contact"
}
},
{
"@type": "WebSite",
"@id": "https://yoursite.com/#website",
"url": "https://yoursite.com",
"name": "Your Company",
"description": "What your site does, in one sentence.",
"publisher": { "@id": "https://yoursite.com/#organization" },
"potentialAction": {
"@type": "SearchAction",
"target": {
"@type": "EntryPoint",
"urlTemplate": "https://yoursite.com/search?q={search_term_string}"
},
"query-input": "required name=search_term_string"
}
}
]
}
</script>
The @graph structure lets you declare multiple entities in one block and cross-reference them with @id. This is how you tell a model "this organization publishes this website."
FAQ Schema
If you have FAQ pages or pages that answer questions, add this on the relevant page:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "What is AI search readiness?",
"acceptedAnswer": {
"@type": "Answer",
"text": "AI search readiness is the degree to which a website can be discovered, understood, and cited by generative AI systems like ChatGPT, Perplexity, and Google AI Overviews."
}
},
{
"@type": "Question",
"name": "How is GEO different from SEO?",
"acceptedAnswer": {
"@type": "Answer",
"text": "SEO optimizes for ranking positions on a search results page. GEO optimizes for selection and citation by generative models that synthesize answers from multiple sources."
}
}
]
}
</script>
FAQ schema is your direct line to question-answer extraction. When a model sees FAQPage with Question and acceptedAnswer pairs, it can pull those answers verbatim. That's why the 18.1% adoption number matters — it's the schema type most directly tied to getting quoted, and 82% of sites don't have it.
Step 4: Content Structure — BLUF (10 minutes)
The final step is editing, not infrastructure. A model can only quote you cleanly if your content is structured to be quoted.
BLUF: Bottom Line Up Front. Every page should open with a standalone answer to its core question. Details go below. The answer goes first.
Before: Narrative Structure (Hard to Quote)
# How Our API Handles Rate Limiting
When we first launched our API in 2023, we didn't have any rate limiting
in place. After a few incidents where a single client overwhelmed the
auth service, we realized we needed a more robust approach. We
experimented with token bucket algorithms, considered sliding window
loggers, and eventually settled on a fixed window counter approach
combined with exponential backoff. Here's how it works...
[800 more words of context, history, and implementation details]
A model reading this has to synthesize the answer from scattered sentences. It will paraphrase without citing, or skip this source for one that's cleaner.
After: BLUF Structure (Easy to Quote)
# How Our API Handles Rate Limiting
Our API enforces rate limiting using a **fixed window counter** with
exponential backoff. The default limit is **100 requests per minute**
per API key. When the limit is exceeded, the API returns HTTP 429 with
a `Retry-After` header indicating the wait time in seconds.
## How It Works
Rate limits are calculated per API key, not per IP address. Each
request increments a counter that resets at the start of each
60-second window. When the counter exceeds 100, subsequent requests
receive a 429 response until the window resets.
## Handling 429 Responses
Clients should implement exponential backoff: wait 1 second before
the first retry, then double the wait on each subsequent retry, up to
a maximum of 60 seconds. The `Retry-After` header provides the exact
wait time for the current window.
The first paragraph is self-contained. A model can extract it as a direct quote. It makes sense outside the context of the full page. That's quotability.
The Editing Rule
Go to your top 5 most important pages. For each one:
- Find the core question the page answers
- Write a 2-3 sentence answer that stands alone
- Put it at the top, before any context or history
- Move the details below
This is the step that takes the most effort but has the highest impact on whether you get quoted or paraphrased.
The Order Matters
I'll say it one more time because it's the most common mistake:
Access → Orientation → Understanding → Quotability
If your robots.txt blocks GPTBot, your llms.txt and schema are invisible. If your llms.txt is broken, your schema sends a model to dead ends. If your schema is missing, your content structure doesn't help the model understand what it's reading.
Fix the foundation first. Each layer enables the next.
What This Gets You
From the benchmark data:
| Signal | With | Without | Gap |
|---|---|---|---|
| llms.txt | 63.3 | 43.3 | +20.0 pts |
| Schema | 61.0 | 32.8 | +28.2 pts |
Sites with llms.txt score 20 points higher than sites without. Sites with schema score 28 points higher. The layers compound.
None of this requires a marketing agency. It requires 30 minutes and a text editor. The four files you need to touch:
/robots.txt/llms.txt- Your homepage
<head>(JSON-LD) - Your top 5 pages (content structure)
Go fix them.
