A service deploys perfectly clean. Pods Running, Endpoints populated, the Certificate resource says Ready: True, the IngressRoute shows no events. Then you curl the hostname and get back a 404 page rendered by your SSO provider, which you never attached to that route.
Nothing in the deployment failed. Traefik logged nothing at error level. The 404 came from a service in a completely different namespace.
That's a host collision, and the mechanism behind it is one of the least intuitive parts of Traefik's routing model.
What you'd expect
Two routers, two rules. One says Host(\agents.example.com), the other says HostRegexp(\^.+.example.com$). A request for agents.example.com matches both. Any sane router picks the more specific match, because that's how routing works basically everywhere else: longest-prefix wins in IP routing, most-specific selector wins in CSS, exact match beats wildcard in DNS.
So the exact Host rule should win. It's narrower. It names one hostname. The regex names an infinite set.
Traefik does not work that way.
What actually happens
Traefik assigns every router a priority. If you don't set one, the default priority is the length of the rule string in characters. That's it. Not specificity, not match type, not creation order. Character count.
Do the arithmetic on those two rules:
Host(`agents.example.com`) → 26 characters
HostRegexp(`^.+\.example\.com$`) → 32 characters
The wildcard is 6 characters longer, so the wildcard wins. Every request for agents.example.com gets handed to whatever service the regex router points at, which in most clusters is an auth proxy sitting in front of everything. The auth proxy gets a Host header for an app it has no provider configured for, and returns its own 404.
The more generic rule outranks the more specific one because it happens to be typed with more characters. Rename your regex to something terser and the winner flips. Add a hyphen to a subdomain and the winner flips back. This is deterministic, it's documented, and it still surprises people every single time.
It gets worse when the tie is exact. Two IngressRoute objects in different namespaces with byte-identical Host rules produce two routers with identical priority. Traefik's Kubernetes CRD provider derives router names from a namespace/name hash, so they don't collide by name and nothing errors out. You get two valid routers competing at the same rank, and the tie-break isn't something you want to build a production dependency on. The winner can change on the next config reload, which is the worst possible failure mode: works on Tuesday, breaks on Thursday, nothing in Git changed.
The auth-proxy fallthrough diagnostic
Here's the pattern worth memorizing, because it saves an hour every time:
If you get a 404 (or a login redirect) from your auth provider on a service that was never wired to that auth provider, you have a Traefik routing collision.
The auth proxy is almost always the shadowing router, for two reasons. It usually owns the broadest rule in the cluster (a wildcard or regex covering the whole domain), and broad rules tend to be long rules. Regex syntax is verbose. HostRegexp is 10 characters before you've matched anything.
The symptom points at the auth stack, so that's where people start debugging. They check the outpost, the provider, the application binding, the token audience. All of it is fine. The auth proxy is behaving correctly given a Host header it doesn't recognize.
Reading the actual routing table
Stop guessing and ask Traefik what it decided. The runtime API exposes every router with its computed priority:
# Port-forward whatever entryPoint serves your API/dashboard
kubectl -n traefik port-forward deploy/traefik 8080:8080
curl -s localhost:8080/api/http/routers \
| jq -r '.[] | select(.rule | test("example\\.com"))
| [.priority, .status, .service, .rule] | @tsv' \
| sort -rn
Sorted descending by priority, the first row is the router that wins. If that row isn't the one you deployed, you've found it in about fifteen seconds. The status field matters too: a router with status: "disabled" won't serve traffic, and that's a separate failure I'll get to below.
Two things this view gives you that the logs don't. It shows the computed priority, including the default length-based value you never wrote down anywhere. And it shows routers from every namespace at once, which is exactly the visibility you lose when you're reading one IngressRoute manifest at a time.
The fix
Set an explicit priority
The IngressRoute CRD takes a priority on each route. Setting it disables the length-based default for that router.
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: agents
namespace: dev
spec:
entryPoints: [websecure]
routes:
- kind: Rule
match: Host(`agents.example.com`)
priority: 100 # beats any length-derived default in practice
services:
- name: agents
port: 8080
tls:
secretName: agents-tls
Pick a band and stick to it. Something like: exact-host routes get priority: 100, wildcard/catch-all routes get priority: 10. Two numbers, written into your chart defaults, and the entire class of bug disappears. Any exact host beats any wildcard regardless of how long anyone's rule string is.
The reason this works better than the alternatives is that it encodes intent. priority: 100 in a manifest says "this is a specific route and it should win." A 45-character rule string says nothing to the next person reading it.
Or narrow the wildcard
If your auth proxy genuinely needs a catch-all, scope it so it can't swallow namespaces it has no business serving. Instead of matching the whole apex domain, match only the subdomains you actually front:
# Broad — catches everything under the domain, including new services
match: HostRegexp(`^.+\.example\.com$`)
# Narrow — catches only what you opted in
match: Host(`sso.example.com`) || Host(`admin.example.com`)
Worth calling out: Traefik v3 changed HostRegexp syntax. The v2 named-group form ({subdomain:[a-z]+}.example.com) is gone; v3 expects standard Go regexp. If you migrated a v2 config and your wildcard silently stopped matching what you thought it matched, that's why. You can set syntax: v2 on a route as a compatibility escape hatch, but treat it as a migration step, not a destination.
The hack that works and that you shouldn't ship
Because priority is string length, you can win a collision by making your rule longer:
match: Host(`agents.example.com`) && PathPrefix(`/`) # 45 characters
PathPrefix(\/) matches everything, so the semantics are unchanged, and you've bought 19 characters of priority. It works. I've seen it in production charts more than once.
Don't do it. It's a magic incantation that breaks the moment someone shortens the hostname or lengthens the competing rule, and nobody reviewing the diff will understand what it's for. Use priority.
The second path to the same symptom: cross-namespace middleware
There's a failure that looks identical from the outside but has nothing to do with rule length.
Since v3, the Kubernetes CRD provider defaults allowCrossNamespace to false. An IngressRoute in the dev namespace referencing a middleware in the auth namespace gets rejected:
routes:
- kind: Rule
match: Host(`agents.example.com`)
middlewares:
- name: forward-auth
namespace: auth # rejected unless allowCrossNamespace: true
Traefik logs a cross-namespace reference error and marks the router as disabled. It does not fail the deployment, does not update the IngressRoute status in a way that kubectl get makes obvious, and does not stop serving traffic. Your route simply isn't in the routing table anymore, so the request falls through to the next-best match. Which is the wildcard. Which is the auth proxy. Which returns a 404.
Same symptom, different root cause, and the /api/http/routers dump distinguishes them instantly: a collision shows your router present with a lower priority, a cross-namespace rejection shows your router disabled or missing entirely.
You can flip allowCrossNamespace: true in the provider config, but understand what you're accepting: any namespace can then attach any middleware from any other namespace, including auth middlewares it wasn't meant to have and, more interestingly, not attach ones it was. That's a real boundary, and turning it off to fix one route is the kind of decision that looks cheap now and expensive later. The same reasoning applies to how you scope identities across routing domains, which I got into in two-tier service accounts for agent workflows.
The double-blind: when the shadowing service is also broken
The nastiest version of this is when the auth proxy that's swallowing your traffic is itself misconfigured. Now you're debugging two failures at once and the symptoms interleave.
A common one: the auth provider's secret key contains a trailing newline. echo "value" | base64 adds one. So does most copy-paste out of a terminal. The provider starts, serves a page, and then fails signature validation on every token, so you get a 404 or a redirect loop that looks exactly like a routing problem.
Check secret shape without ever printing the value:
NS=auth; SECRET=provider-config; KEY=SECRET_KEY
raw=$(kubectl -n "$NS" get secret "$SECRET" \
-o jsonpath="{.data.$KEY}" | base64 -d | wc -c)
clean=$(kubectl -n "$NS" get secret "$SECRET" \
-o jsonpath="{.data.$KEY}" | base64 -d | tr -d '\n\r' | wc -c)
[ "$raw" -eq "$clean" ] \
&& echo "OK: no stray newlines ($raw bytes)" \
|| echo "FAIL: $((raw - clean)) newline/CR byte(s) in $KEY"
The value passes through the pipe and never lands in a variable or on stdout. Only the byte counts do. Run it as a pre-flight check on every secret that feeds an auth stack; the two-second version saves you from a debugging session where nothing makes sense.
The collision-free pattern for internal services
Not every service needs to sit behind the global auth wrapper, and routing everything through one broad rule is what creates the collision surface in the first place. For internal-only services, an IP allowlist plus a NetworkPolicy gives you a route that can't be shadowed because it never overlaps with anything:
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: lan-only
namespace: dev
spec:
ipAllowList:
sourceRange:
- 10.0.0.0/16
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: agents
namespace: dev
spec:
entryPoints: [websecure]
routes:
- kind: Rule
match: Host(`agents.example.com`)
priority: 100
middlewares:
- name: lan-only # same namespace, no cross-namespace friction
services:
- name: agents
port: 8080
tls:
secretName: agents-tls
Middleware lives in the same namespace as the route, so allowCrossNamespace is irrelevant. Priority is explicit, so rule-length arithmetic is irrelevant. Pair it with a default-deny NetworkPolicy at the pod level, the way I laid out in default-deny and namespace isolation with Calico, and you've got defense in depth without a single shared routing rule.
The tradeoff is honest: you're trading centralized auth for per-service configuration. More YAML, more places to get it wrong, no single place to revoke access. For a public-facing app, the global wrapper is the right call. For an internal dashboard, the allowlist wins on blast radius alone. This kind of routing-boundary decision is one of the things I spend a lot of time on in infrastructure consulting work, and the answer really does change per service.
Catching it before merge
Host collisions are a static property of your manifests. You don't need a running cluster to find them, which makes them a good fit for the kind of CI validation I described in catching broken YAML before merge:
# Fail the build if two IngressRoutes claim the same exact Host
grep -rhoE 'Host\(`[^`]+`\)' --include='*.yaml' ./manifests \
| sort | uniq -d | grep . && {
echo "duplicate Host rules found"; exit 1; }
Crude, and it won't catch regex overlaps, but it catches the exact-duplicate case that produces the nondeterministic tie. Extend it to flag any HostRegexp that lacks an explicit priority, and you've covered the two ways this actually bites.
What I'd change
Set priority on every route from day one. It costs one line and it removes an entire failure class. The default length-based behavior is fine for a single-tenant setup with five services; it stops being fine the moment two teams write rules against the same domain and neither one knows the other exists.
Treat the auth-provider 404 as a routing signal, not an auth signal. That reflex alone cuts debugging time dramatically, because it redirects you from the auth stack (where everything is working) to the routing table (where the answer is).
And keep /api/http/routers in your muscle memory. Manifests describe intent. The runtime API describes what Traefik actually built, priorities included, across every namespace at once. When those two disagree, the runtime API is right and your mental model is wrong. If you're also chasing certificate mismatches on the same hostnames, wildcard DNS and ndots:5 covers the other half of that puzzle, and it interacts with this one more often than you'd like.
