ForwardAuth with Authentik: Wiring SSO Across Multiple Apps Without Breaking Ingress

The most misleading thing Authentik will ever show you is its own 404 page.

TLS terminated correctly. Traefik matched the route. The ForwardAuth middleware fired, reached the outpost, and got an answer back. Every layer did its job. And the answer was a branded 404, because the proxy provider was never bound to the outpost that answered. Nothing is broken in a way that shows up in logs, which is exactly why people burn an afternoon on DNS before they check the outpost binding.

Who this is for

You're running Traefik as your ingress controller on Kubernetes, you have more than two internal apps that need auth, and you'd rather not implement OIDC in each of them. ForwardAuth is the right answer for that shape of problem: one identity provider, one middleware, N apps that stay blissfully unaware they're behind SSO.

The install is thirty minutes. The operational hardening is the part nobody writes down, so that's what this covers: the failure modes that look like working systems, and the configuration that survives a default-deny cluster and a GitOps controller that reconciles your ingress every three minutes.

The failure modes that look like success

1. The branded 404

When Traefik forwards an auth check to the Authentik outpost and the outpost has no provider bound for that hostname, it doesn't return 401 or 403. It returns a 404 from its own web UI, styled with your Authentik branding. Traefik dutifully passes that through, and the user sees a polished "not found" page for an app that is definitely running.

The diagnostic that actually settles it:

# Ask the outpost directly, pretending to be Traefik.
kubectl -n traefik run curl --rm -it --image=curlimages/curl --restart=Never -- \
  curl -sS -o /dev/null -w '%{http_code}\n' \
  -H 'X-Forwarded-Proto: https' \
  -H 'X-Forwarded-Host: wiki.example.com' \
  http://authentik-server.authentik.svc.cluster.local/outpost.goauthentik.io/auth/traefik

A healthy unauthenticated check returns 401. A 404 means the outpost has no provider matching X-Forwarded-Host. A 302 means it's redirecting you to the login flow, which is also fine. Anything else means the request never reached the outpost at all.

The fix is in the Authentik UI, not in your manifests: the proxy provider must be attached to an application, and that application must be selected in the outpost's list. Creating the provider is not enough. Creating the application is not enough. The outpost holds an explicit list, and a provider that isn't on it does not exist as far as the auth check is concerned.

2. The secret key with a newline in it

Generate the Authentik secret key with openssl rand -base64 60 and you get 80 base64 characters, which openssl wraps across two lines. Store that in a Kubernetes secret and the trailing line feed comes along for the ride.

The embedded outpost uses that value when it talks back to the Authentik core API, in an Authorization header. Go's net/http validates header values before it writes them to the wire and rejects anything containing a control character. The request fails inside the client library, never leaves the pod, and you get:

http: invalid header field value for "Authorization"

The outpost then reports zero providers loaded, which puts you right back at the branded 404. Two different root causes, one identical symptom.

Generate it in a shape that can't wrap:

# Hex output is single-line at any length. No wrapping, no tr, no surprises.
openssl rand -hex 48

# If you want base64, strip line endings explicitly:
openssl rand -base64 60 | tr -d '\r\n'

Then verify what actually landed in the pod, without ever printing the value:

kubectl -n authentik exec deploy/authentik-server -- sh -c '
  printf "%s" "$AUTHENTIK_SECRET_KEY" | wc -c
  printf "%s" "$AUTHENTIK_SECRET_KEY" | tr -dc "[:cntrl:]" | wc -c
'

First number is the length. Second number must be 0. If it's 1, you have a trailing newline. The tr -dc deletes everything that isn't a control character and counts what's left, so the secret itself never reaches your terminal or your shell history. Same discipline applies when you seal it: if you're storing this in Git via SealedSecrets, pipe the generator directly into kubectl create secret --dry-run=client rather than round-tripping through a file that your editor will happily terminate with a newline.

3. Default-deny eats the auth check

If you run Calico NetworkPolicies with default-deny, ForwardAuth introduces a traffic path you probably didn't account for: ingress controller to identity provider, on an internal port, in a different namespace. Most SSO tutorials assume a wide-open cluster.

What makes this one nasty is the timing. Traefik's ForwardAuth has a default timeout, so a blocked packet doesn't produce a clean error. It produces a slow 500 after the connection times out, and the Traefik access log records a request that took several seconds and failed at the middleware. Under load it looks like the identity provider is overwhelmed rather than unreachable.

You need policy on both ends. Egress from Traefik:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: traefik-to-authentik
  namespace: traefik
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: traefik
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: authentik
      ports:
        - protocol: TCP
          port: 9000

And matching ingress on the Authentik side, selecting the outpost pods. If you're using the embedded outpost, that's the authentik-server deployment; a dedicated outpost gets its own ak-outpost-* pods. Do not forget DNS egress to kube-system in the same policy set, or the service name won't even resolve.

4. Authenticated, and still a guest

This is the one that survives all the way to production. SSO works, the login page appears, the redirect completes, the app loads. And the user is anonymous inside the application.

ForwardAuth gives Traefik a set of headers from the outpost, but Traefik only copies the ones you explicitly list. The default is none. So the app receives an authenticated request with no identity attached and falls back to its guest role.

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: authentik-forwardauth
  namespace: traefik
spec:
  forwardAuth:
    address: http://authentik-server.authentik.svc.cluster.local/outpost.goauthentik.io/auth/traefik
    trustForwardHeader: true
    authResponseHeaders:
      - X-authentik-username
      - X-authentik-groups
      - X-authentik-email
      - X-authentik-name
      - X-authentik-uid
      - X-authentik-jwt

Then the app has to be told to trust those headers, and that's per-application work that no amount of ingress configuration will do for you. Grafana wants auth.proxy enabled with header_name = X-authentik-username and header_property = username. Wiki.js needs its header authentication strategy configured with the group header mapped to Wiki.js groups. Some apps want the email address as the identifier rather than the username, which matters the first time someone changes their display name and gets a brand new account.

X-authentik-groups is a pipe-separated list, not JSON and not comma-separated. If your app's group mapping silently produces one giant group named admins|editors|viewers, that's why.

The multi-app pattern

Once you're past two applications, the per-app setup starts to hurt. Single-application forward auth requires routing /outpost.goauthentik.io/ on every app's hostname back to the outpost, because that's where the OAuth callback lands. Miss it on one app and users get a 404 immediately after logging in, on the redirect back.

Domain-level forward auth collapses that. One proxy provider covers *.example.com, one auth subdomain handles all callbacks, one middleware gets attached everywhere.

That third row is the tradeoff, and it's the reason domain-level isn't strictly better. With domain-level forward auth, the outpost checks that the session is valid for the domain, not that this specific user is authorized for this specific app. Anyone who can log into one application can reach all of them. Authentik's docs are explicit about this and people still miss it, then wire up per-app policy bindings that quietly do nothing.

For a homelab or a small team where everyone gets everything, domain-level is the right call. For anything with real access tiers, use single-application providers and pay the per-app routing cost, or use domain-level for the perimeter and enforce authorization inside each app using the group headers.

Domain-level provider settings that matter:

• External host: https://auth.example.com, the browser-reachable URL. Not the cluster service DNS. The outpost puts this in redirect URLs, and the browser has to follow them.
• Cookie domain: example.com. Set this to the parent domain or the session won't carry between subdomains, which defeats the entire point.
• Token validity: shorter than you think. The session cookie is your blast radius.

The authentik_host mistake is worth its own sentence. Setting it to the internal service name produces a system that works perfectly for the auth check and then redirects the user's browser to a hostname that only resolves inside the cluster. The login page never loads and the address bar shows something like http://authentik-server.authentik.svc.cluster.local/..., which is a good clue and an easy one to misread as a DNS problem.

Attaching the middleware without breaking routing

Traefik v3 disallows cross-namespace middleware references by default. An IngressRoute in the apps namespace cannot reference a Middleware in the traefik namespace unless you set providers.kubernetesCRD.allowCrossNamespace=true in the static configuration.

Two ways out. Enable cross-namespace references and keep one canonical middleware, or replicate the middleware into every namespace that needs it. I prefer the single canonical middleware: a duplicated auth config is a config that will drift, and the one that drifts will be the one protecting something you care about.

With cross-namespace enabled, an IngressRoute references it by namespace-name@kubernetescrd:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: wiki
  namespace: apps
spec:
  entryPoints: [websecure]
  routes:
    - match: Host(`wiki.example.com`)
      kind: Rule
      middlewares:
        - name: authentik-forwardauth
          namespace: traefik
        - name: private-range
          namespace: traefik
      services:
        - name: wiki
          port: 80
  tls:
    secretName: wiki-tls

Order matters. Middlewares run in the order listed, so an IP allow-list in front of the auth check means unauthorized networks never reach the identity provider at all. Cheap, and it keeps your outpost logs readable.

If you use plain Ingress objects with annotations instead, the reference is a comma-separated string:

annotations:
  traefik.ingress.kubernetes.io/router.middlewares: traefik-authentik-forwardauth@kubernetescrd

Note the format: namespace and name joined by a hyphen, then @kubernetescrd. A typo here doesn't produce an error page. Traefik logs a warning and skips creating the router, and you get a 404 with no obvious connection to the annotation.

When Authentik lives outside the cluster

If your identity provider runs on a VM or an LXC rather than in the cluster, the instinct is an ExternalName service. Traefik handles those poorly for ForwardAuth targets, and you also lose the ability to write NetworkPolicy against them, because there's no pod and no IP to select.

A selector-less Service with a manually managed EndpointSlice gives you a normal ClusterIP that policy can reason about:

apiVersion: v1
kind: Service
metadata:
  name: authentik-external
  namespace: authentik
spec:
  ports:
    - name: http
      port: 80
      targetPort: 9000
apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: authentik-external-1
  namespace: authentik
  labels:
    kubernetes.io/service-name: authentik-external
addressType: IPv4
ports:
  - name: http
    port: 9000
endpoints:
  - addresses: ["10.0.0.50"]   # the external Authentik host

The label linking the slice to the service is mandatory and unlabelled slices are ignored without complaint. Use EndpointSlice rather than the older Endpoints API; it's the supported path going forward and the manual-endpoints pattern is one of the few places where you still hand-write these.

Why the pieces fit together this way

ForwardAuth is a subrequest. When a request arrives, Traefik pauses it and issues a separate HTTP request to the auth address, carrying X-Forwarded-Proto, X-Forwarded-Host, X-Forwarded-Uri, and the original cookies. The outpost's entire job is to answer with a status code.

2xx means proceed, and Traefik continues to the backend with whichever response headers you listed in authResponseHeaders copied onto the upstream request. Anything else, and Traefik returns the outpost's response to the client verbatim, including its body and its Location header. That verbatim passthrough is the mechanism behind the branded 404: the outpost genuinely returned 404, so the user genuinely sees 404. Traefik is not hiding an error, it's relaying one faithfully.

trustForwardHeader: true is what lets the outpost use the X-Forwarded-* values to figure out which application is being requested. Without it, every request looks like it's for the outpost's own hostname, and provider matching fails for everything.

The redirect chain is why external_host has to be publicly resolvable. On an unauthenticated request, the outpost answers 302 with a Location pointing at the login flow. Traefik relays that to the browser. The browser follows it. At that moment the resolution happens on the user's machine, on their network, with their DNS. Cluster-internal names are meaningless there. If you're already running split-horizon DNS with AdGuard Home, point the auth hostname at your load balancer IP internally and make sure the certificate covers it, because a TLS error on the auth redirect looks identical to an auth failure from the user's side.

Guarding against drift

GitOps introduces a specific hazard here: the auth middleware is an annotation or a list entry on an object that other tooling also manages. A Helm chart upgrade that regenerates the Ingress template drops your annotation. ArgoCD reconciles it. The app is now public, and the only visible change is that it stopped asking people to log in, which almost nobody reports as a bug.

An IngressRoute referencing a deleted Middleware fails closed, returning 503, and that's the behavior you want. An Ingress whose middleware annotation was removed fails open. If you're managing this through ArgoCD app-of-apps, prefer IngressRoute for anything behind auth for exactly that reason.

Belt and suspenders is an admission policy. A Kyverno rule that rejects any Ingress or IngressRoute in a labelled namespace unless it carries the auth middleware turns a silent exposure into a failed sync you'll actually see. That's a ten-line policy protecting against a class of mistake that no amount of care prevents, because the mistake is made by a templating engine, not a person.

What I'd tell someone starting this

Check the outpost binding before you check anything else. The provider-to-application-to-outpost chain has three links and breaking any of them produces the same 404. It's the highest-probability cause and the fastest thing to verify.

Curl the outpost from inside the Traefik namespace as your first diagnostic, not your fifth. A 401 from that call means the auth layer is fine and your problem is somewhere else entirely, which eliminates most of the search space in one command.

Generate secrets in shapes that can't wrap, and verify the byte count in the pod rather than trusting the generator. Control characters in a config value produce errors that name a completely unrelated component, and you'll read that error five times before you suspect the secret.

Decide between domain-level and per-app providers based on your authorization model, not your convenience. Retrofitting per-app authorization onto a domain-level deployment means rebuilding the provider layer while people are using it.

Write the NetworkPolicy in the same commit as the middleware. Adding ForwardAuth creates a new east-west traffic path, and in a default-deny cluster that path is closed until you open it. Discovering this during a partial rollout is a lot less pleasant than discovering it in a diff.

The broader theme is that identity in Kubernetes is a chain, and chains fail at whichever link you didn't configure: ingress routing, the auth subrequest, header propagation, application-side mapping, and the RBAC or group model behind it. A working login page tells you about link two. It tells you nothing about link four, which is where users end up as guests. If you're building out this kind of identity plumbing across a real environment and want a second set of eyes on the design, that's the kind of infrastructure work I do.

Test the whole chain with a real user account in a real browser, in an incognito window, from a network that isn't your workstation. Every shortcut around that has a way of hiding exactly the link that's broken.