I run a maintenance sweep over my own site every day. Yesterday it found 1,474 dead internal links that had been sitting there in plain sight, on 737 pages, and every automated check I had was green.
The cause is a bug class I have not seen written up, so here it is.
Two places encode "which languages does this page have"
A partially translated site has, whether you planned it or not, two independent answers to the question does this page exist in Chinese?
- The
<link rel="alternate" hreflang="...">tags in<head>— what you tell crawlers. - The EN / 简 / 繁 buttons in your nav — what you tell humans.
Mine disagreed. The hreflang side was correct: a few page families on my site are English-only, so those pages emit an English self-reference and nothing else. There is even a comment in the source explaining why, because declaring an alternate that 404s poisons the whole hreflang cluster.
The nav switcher never got that memo. It was doing the naive thing:
// every page, unconditionally
`<a href="${basePath}">EN</a>
<a href="/zh-cn${basePath}">简</a>
<a href="/zh-tw${basePath}">繁</a>`
So on every English-only page, the two Chinese buttons pointed at URLs that do not exist. Click 简 on the API docs, get a 404.
Why nothing caught it
This is the part worth internalising. The bug survived because it fails every test you would think to write:
- The pages are all 200. Uptime checks, sitemap validation, status sweeps — all green. The broken thing is a link on a healthy page.
-
The
hreflangis correct. Any SEO audit that checks hreflang reciprocity passes. - The 404s are correct too. Those Chinese URLs should 404; the page genuinely does not exist in Chinese.
- It is invisible on the pages you look at most. My homepage, tools, and guides are translated, so the switcher works perfectly everywhere I click during normal use. It only breaks in the families I never browse by hand.
Every component was individually correct. The defect only existed in the relationship between two of them.
The check that found it
Nothing clever. Take a page, extract every on-site href, and curl all of them — recording the status and the redirect count:
curl -s https://example.com/some-page \
| grep -o 'href="/[^"]*"' | sed 's/href="//;s/"//' | sort -u \
| while read -r p; do
code=$(curl -s -o /dev/null -w '%{http_code}:%{num_redirects}' "https://example.com$p")
[ "$code" = "200:0" ] || echo "$p -> $code"
done
The num_redirects half matters as much as the status. 200:1 is a link that works but burns a hop, and a page full of those is a page slowly leaking crawl budget. I run this against one rotating page per sweep rather than crawling the whole site — cheap, and over a couple of weeks it covers everything that matters.
On the page I happened to rotate to yesterday, the API docs, it printed exactly two lines. Both were the language switcher.
The fix is not "add the missing pages"
The tempting fix is to generate the Chinese versions so the links resolve. That is how you end up maintaining machine-translated pages nobody reads, and — if they serve near-identical content — competing with yourself in search.
The real fix is to make the two sources of truth one source of truth. I already had a set listing the English-only paths, used by the hreflang generator. The switcher just needed to consult it:
const zhLinks = EN_ONLY_PAGES.has(basePath) ? '' : `
<a href="/zh-cn${basePath}">简</a>
<a href="/zh-tw${basePath}">繁</a>`;
On English-only pages the switcher now renders the EN button alone. 1,474 dead links gone; the rest of the site keeps its full switcher. Whenever a page family gets translated for real, adding it to that one set fixes the crawler signal and the human UI in the same commit — which is the actual win here.
Three things to take away
- If two subsystems answer the same question, one of them is going to be wrong. Language availability, feature flags, pricing tiers — anywhere you have a "which of these exist" list, count how many places encode it.
- Green status checks do not mean green links. A 200 page can be full of 404s. Check the edges of your graph, not just the nodes.
- Audit the family, not a sample. Spot-checking would have found nothing, because the switcher works on most pages. Enumerating every link on one page found it in seconds.
The crawler-facing side of this is what I care about most, since AI assistants read my machine-readable site directory and follow whatever URLs it hands them. A dead link is not just a bad click — it is a citation you never get.
Go curl your own language switcher.
