You are about to launch a new site and the advice keeps multiplying: add /ai/summary.json, wire up llms.txt, sprinkle FAQPage markup, build a discovery layer. But which of these does an AI crawler actually care about?
We run a Cloudflare Worker that logs every request to geo010.com and tags it by crawler, so we could answer that question with data instead of opinion. This article attributes 15 days of AI crawler traffic to nine GEO measures, shows which ones changed nothing, and turns the results into a launch checklist for a new site.
How Did We Attribute Traffic to Each GEO Measure?
Every request to geo010.com passes through a Worker that records the date, path, status, page type and a normalized crawler name into a D1 database. For each of nine GEO measures we shipped, we then asked the logs three questions: which crawlers hit the relevant paths, how often, and did visits change after the measure shipped?
Two warnings before the numbers. First, a crawler label is only a user agent — attackers spoof them, so we cross-checked paths and status before counting anything. Second, some "visits" were us: we verify deployments with curl, and our own checks show up in the logs. We filtered those out.
Which GEO Measures Actually Drove Crawler Traffic?
Robots.txt and sitemap.xml were the only paths every major AI crawler used. The homepage was the only page they all visited. Everything else showed up in a minority of crawlers — or none at all.
Here is the full attribution table, ranked by how much real crawler traffic each measure produced:
GEO measure Which crawlers Evidence (Aug 4–19) Impact
robots.txt ClaudeBot, Googlebot, OAI-SearchBot, Bingbot, FacebookBot ClaudeBot 104, Googlebot 32, OAI-SearchBot 26, Bingbot 17 Highest — nearly daily entry point
sitemap.xml ClaudeBot, Bingbot, GPTBot ClaudeBot 104, Bingbot 11, GPTBot 9 Highest — ClaudeBot reads it religiously
Homepage / GPTBot, PerplexityBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Applebot GPTBot 21, PerplexityBot 12, OAI-SearchBot 10, ClaudeBot 8 High — visited by every major crawler daily
New content pages GPTBot (primary), ClaudeBot, PerplexityBot GPTBot 121 content pages; ClaudeBot 2 community articles High — GPTBot traverses all pages
Freshness alignment All (indirect) Crawler bursts on 08-08, 08-12 and 08-17 followed freshness passes Medium — sustains re-visits
llms.txt GPTBot, Amazonbot, Googlebot 4 total requests, GPTBot 2 Low — only GPTBot treats it as an entry
Definition-first layout GPTBot, ClaudeBot, Bingbot, others Entity-density page reached by 7+ crawlers Low-medium — quality, not quantity
FAQPage markup None (AI) 0 visits from major AI crawlers after shipping None (crawl) — a citation-layer lever
AI discovery endpoints (/ai/, ai.txt, plugins) None 0 requests in 15 days None — not used by today's crawlers
The ranking surprised us. The "boring" infrastructure files — robots.txt and sitemap.xml — produced more genuine crawler traffic than every shiny AI feature combined.
Why Do GPTBot and ClaudeBot Crawl Completely Differently?
The two most important AI crawlers could not be more different in behavior, and the difference determines what to build first:
Behavior GPTBot ClaudeBot
Reads robots.txt Rarely (1 request) Religiously (104 requests)
Reads sitemap.xml Occasionally (9) Every day (104)
Visits content pages Yes — 121 pages, including a full-site pass on Aug 17 Rarely (a handful total)
Entry strategy Traverses the whole site per pass Follows robots → sitemap, then mostly home
ClaudeBot is a disciplined path-follower. It checks your robots and your sitemap almost daily, but it barely descends into content unless something in that sitemap or homepage directs it. GPTBot is a deep scanner. It skips robots, reads llms.txt occasionally, and walks the entire site — on Aug 17 it made a single pass touching 60 pages, each one once.
The practical consequence: if your robots.txt or sitemap.xml is broken, you hurt the crawler that reads them daily (ClaudeBot). If your content is shallow, you waste the crawler that reads everything (GPTBot).
What Did We Get Wrong in Earlier Analyses?
Honesty check. Earlier write-ups on this site credited FAQPage and new-article launches with immediate crawler spikes. Our per-request attribution showed those were misattributed:
"A new article got crawled 6-10 times on launch day." Actually most of those hits were browser visitors and our own checks; each AI crawler visited the new page once. New content is crawled once per GPTBot pass, not burst-spiked by the act of publishing.
"FAQPage pages were visited by ClaudeBot and OAI-SearchBot the same day." Not supported by the logs — the six pages got no visits from major AI crawlers after the markup shipped. Their visitors were browsers, Bingbot and our curl checks.
"The 08-11 traffic spike was AI growth." It was a scanner attack — ChatGPT-User and Amazonbot probing /.env.production and /telescope/requests. Same for a spoofed-GPTBot burst on 08-13.
None of those measures are useless. They just do their work at a different layer: FAQPage helps a page be quoted once AI is already considering it; llms.txt helps GPTBot orient; discovery endpoints may help future tooling. The mistake was measuring them against crawl counts at all.
What Three More Days Confirmed
When we first published this article, the data window was 12 days. Three more days (Aug 17–19) gave us a live test of the same predictions:
Date What happened What it proved
Aug 17 GPTBot hit 60 pages in a single pass — the first full-site traversal since launch Content pages are discovered via sitemap, not via AI endpoints. GPTBot never touched /ai/summary.json.
Aug 17 ClaudeBot made 23 requests: all robots.txt + sitemap.xml + homepage ClaudeBot has not changed its entry-only behavior in three weeks. Sitemap quality matters more than content quantity for this crawler.
Aug 17 Total requests 1,066; only 371 returned 200 Scanner noise persists — the same path-spam pattern (old PHP paths, /api/v1/validate/code) returns daily. Noise filtering is ongoing work.
Aug 18 Bingbot made 51 requests (highest single-day for Bing); Meta-ExternalAgent appeared for the first time with 8 requests Not all crawlers behave the same. Bingbot probes old/cached paths; Meta-ExternalAgent is a new entrant worth monitoring.
Aug 18–19 Both community articles (the crawler-data post and this one) each got 1 GPTBot visit + 1 ClaudeBot visit New content is indexed once per crawler type — not burst-on-publish. The "crawled 6-10 times on launch day" claim from earlier articles was definitely noise.
The extra data made the picture sharper, not murkier: robots + sitemap + homepage are the crawl foundation; everything else is a citation-layer optimization you add later.
What Should a New Site Build First?
If we were launching a new site today, we would build three things before anything else: a correct robots.txt, a sitemap.xml where every URL returns 200, and a server-rendered homepage that explains what the site is — plus a few genuinely useful articles.
Make robots.txt correct. Allow the AI crawlers you want, block only what you must (analytics, staging, private pages). A wrong rule that blocks everything is a silent zero.
Make sitemap.xml 100% valid. Every URL in it must return 200. A sitemap full of 404s teaches the daily visitor (ClaudeBot) that your site is broken.
Serve real HTML on the homepage. AI crawlers do not run JavaScript. The homepage must contain the answer to "what is this site?" in plain text, with links to your best pages.
Publish 3-5 real articles before launch day. GPTBot traverses everything; give it pages worth reading. Five solid pages beat fifty shells.
That is the entire pre-launch crawl work. It took us the least effort of all nine measures, and it produced the most crawler traffic.
What Should a New Site NOT Build First?
Skip these until the foundations above exist. Our logs show they contribute nothing to crawl, however good they look on a roadmap:
AI discovery endpoints (/ai/summary.json, /.well-known/ai.txt, ai-plugin.json, openapi.json) — zero crawler requests in 15 days. Audit tools like them; crawlers ignore them.
FAQPage markup on every page — no crawl effect in our data. Add it later, on pages that are already candidates for citation.
A monitoring dashboard — start with your server logs or platform analytics for the first week; you only need to know whether crawlers are arriving and what they hit.
Freshness plumbing — worth building, but only once you have content and dates to keep honest.
This does not mean those features are bad. It means they are phase-two work, and building them before the crawl foundation is optimizing a pipeline that is not flowing yet.
FAQ
What should a brand-new site build first for AI crawlers? A correct robots.txt, a sitemap.xml where every URL returns 200, and a server-rendered homepage with a one-line statement of what the site is plus a few real articles. Those were the only entry points every AI crawler used in our logs.
Do AI discovery endpoints like /ai/summary.json help AI crawlers? Not in our 15 days of logs. GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot visited robots.txt and sitemap.xml almost every day but never requested any AI-facing endpoint.
Why do GPTBot and ClaudeBot behave differently? ClaudeBot follows the classic crawl path — robots.txt and sitemap.xml daily, but barely any content. GPTBot ignores robots, reads llms.txt occasionally, and traverses the entire site — 121 content pages touched, including a single full-site pass on Aug 17.
Does adding FAQPage markup get a page crawled more? In our data, no measurable crawl effect. It improves how a page is quoted once AI is already considering it — it is a citation-layer lever, not a crawl driver.
