2,894 Requests Hit My Top Page. Nobody Ever Requested It.

The most-read page last month, on a site I look after, was a URL that nobody ever requested.

I did not set out to find that. I was going to write about Meta.

Meta's crawlers account for 43% of all AI crawler traffic on the site I have been measuring — more than OpenAI, Anthropic and Perplexity combined. That is a strange enough fact to want an explanation, so I opened the panel that lists which pages the crawlers actually read, expecting to see what Meta was so interested in.

The top row was /tag//. Two slashes. 2,894 requests in thirty days, most of them from Meta's crawler, which is more than half of everything that crawler did on the site.

A better story than the one I had

So I loaded it. /tag// returns a 301 to /tag/, and /tag/ returns a 404. A redirect to a dead end.

For about ten minutes this was a much better article than the one I had planned. The largest AI crawler on this site was spending half of its visits hammering a URL that goes nowhere, ninety-six times a day, and the site owner had no way to know. Write that down, publish it, done.

Before writing it I wanted to know where the crawler had found the URL. That is where it fell apart.

• The site's home page contains 29 links to tag pages. Every one of them is a well-formed absolute URL. None has a double slash.
• The sitemap lists 276 tag URLs. None has a double slash, and /tag/ itself is not in it.
• The 301 carries x-redirect-by: WordPress, so PHP is running on every one of those requests. Nothing is being cached or shortcut.

A crawler does not invent URLs out of nothing, and there was nothing here to find it in. Which left one place I had not looked.

The call was coming from inside the house

My own recording code, one line of it:

$path = sanitize_text_field( $path );

sanitize_text_field() is the reflexive way to clean a string in WordPress. It appears in every tutorial and most plugins. Its documentation lists five things it does. Four of them are what you expect: checks for invalid UTF-8, converts a bare < to an entity, strips tags, collapses whitespace.

The fifth is "Strips percent-encoded characters."

Five words, no explanation, and no warning anywhere on the page about what that means for a URL. Here is what it means for a URL:

The site is Japanese. Its tag slugs are Japanese, so in a URL they are percent-encoded, so my sanitiser deleted them. Every Japanese tag page on the site was being written to the database as the same six characters, and then grouped together by a GROUP BY url_path that was doing exactly what it was told.

Meta was not stuck. Meta was reading hundreds of different tag pages, perfectly normally. I was the one turning them into one row.

0

Times anyone requested the URL at the top of my most-read-pages list.
It was not a page. It was 2,894 requests to many different pages, with the distinguishing part deleted.

Who else this happens to

Everyone whose URLs are not ASCII.

A slug in Japanese, Chinese, Korean, Thai, Greek, Hebrew, Arabic, or any Cyrillic language is percent-encoded in a URL. So is a French or Portuguese or Turkish slug with an accent in it. On all of those sites, any tool that runs a path through sanitize_text_field() is storing a path with the identifying part removed — and storing it silently, in a form that still looks like a path.

The function is not broken. Stripping percent sequences is a reasonable defence for a free-text field, where an encoded payload has no business being. It is only catastrophic when the field is percent-encoded on purpose, which is to say whenever it is a URL. The documentation states the behaviour and leaves you to work out the consequence.

Why this kind of bug survives

I had been looking at that dashboard for weeks. So had the site's owner — I manage it, they own it. Neither of us noticed, and I want to be precise about why, because the reason generalises past this one function.

• It never threw anything. No error, no warning, no empty result. A crash on day one would have cost me an afternoon.
• The output was plausible. /tag// looks like a slightly malformed URL, which is a thing that genuinely happens. It reads as a finding, not as a defect. I nearly published it as one.
• It failed by merging, not by losing. No request was dropped. The totals were all correct. Damage that shows up as a smaller number gets noticed; damage that shows up as one confident large number does not.
• It only affects other people's alphabets. Every test I had written used ASCII paths, because I wrote them. On an English site this code is flawless.

That last one is the part I keep turning over. The bug was invisible to me specifically, in a way it would not have been to someone who tested in their own language first.

What to do instead

If you are storing a whole URL, WordPress has esc_url_raw() for it. If you are storing a path and you want it to survive intact, do the work explicitly rather than reaching for the general-purpose cleaner: reject invalid UTF-8, strip control characters, cap the length, and leave the percent sequences alone. That is roughly ten lines, and it is the ten lines I should have written the first time.

Then write a test with a non-ASCII path in it. Mine now has four, and they exist for exactly one reason, which is that I will not remember this in a year.

What this did and did not break

Being specific matters here, because I have published numbers from this instrument twice and I would rather say plainly which ones this touches.

• Broken: which pages were read. On any non-English site, that panel has been unusable. Not subtly wrong — unusable.
• Not broken: how much was read. The path is used in one other place, a check that decides whether a request counts as a content page. That check is a list of exclusions — robots.txt, sitemaps, static file extensions, login and admin endpoints — and every one of them is ASCII. A Japanese path is classified the same before and after the damage.
• Not broken: anything else. Request counts, the split by crawler purpose, the verified percentages: none of them touch the path.

The fix went in on September 6. Old records cannot be recovered — the deleted bytes are gone — so the page list will be correct from that date and wrong before it, which is its own small ongoing embarrassment.

And Meta?

Still 43% of everything, still unexplained, still crawling more of this site than OpenAI, Anthropic and Perplexity put together. I will get to it. It turns out I did not have the instrument to write that article yet, and I would not have known that if the top row of my own dashboard had been slightly less strange.

How to measure your own site

The instrument is a WordPress plugin called AILYS Lens, and it is free. I build it, which by now you have enough evidence to factor in. Every calculation runs on your own server, nothing is transmitted anywhere, and human visitors are never recorded. If you run it on a site whose URLs are not in English, update it — the page list before this fix was telling you nothing.

A note on AILYS Doctor

AILYS Lens is free permanently. That is a design commitment rather than a pricing stage — every feature runs locally on your server, so there is nothing for a paid tier to unlock.

The diagnostic service alongside it, AILYS Doctor, is a different thing. Lens tells you what happened; Doctor tells you why, and what to change. It is free to use during its data-collection period, and any future pricing will be announced on the site.

Updated September 15, 2026: this note originally said Doctor would become a paid subscription on that date. The change has been postponed, and Doctor remains free for now.

See a sample diagnosis · AILYS Doctor

Illustration generated with AI and selected by the author.