A Kubernetes Service stops responding. The first move is always the same: get on the node and run iptables-save | grep KUBE-SERVICES to see which Pod a given Service address maps to.
kube-proxy wrote those rules. It's a standard Kubernetes component that runs on every node and keeps Service forwarding rules registered in iptables. Looking up where a Service routes means reading those rules.
Cilium is a Kubernetes CNI plugin — the piece that wires up Pod networking so Pods can talk to each other. Flannel and Calico fill the same role; you pick one per cluster.
Cilium has a mode that goes beyond Pod networking and also handles Service forwarding, the job kube-proxy used to own. That mode is called kube-proxy replacement. With kubeProxyReplacement=true, Cilium runs on the assumption that kube-proxy is absent. Cilium itself does not remove kube-proxy; on an existing cluster you have to remove it separately. The test environment in this article is built with kind's kubeProxyMode: none, so kube-proxy was never present.
On a node in that Cilium environment, run the same iptables-save | grep KUBE-SERVICES from before. Nothing comes back. The same setup on a kube-proxy cluster returns 13 lines; here it returns 0. KUBE-SERVICES is the chain kube-proxy creates — the entry point for packets destined to a ClusterIP. It doesn't exist.
Broaden the grep to KUBE- and you get a few lines back, but they're KUBE-FIREWALL and KUBE-KUBELET-CANARY, chains created by kubelet that have nothing to do with Service forwarding. The nat table itself has some rules; it isn't empty. Only the Service forwarding rules are missing.
You try a different table, a different node. KUBE-SERVICES is nowhere, and yet the application is up and Services are reachable.
Nothing is broken. The information just isn't in iptables.
This article covers three things: where Cilium stores the forwarding rules that used to live in iptables; why tcpdump on a ClusterIP produces nothing and conntrack -L returns no matches; and what to check, in what order, when connectivity fails.
The test setup runs two kind clusters side by side — one with kube-proxy, one with Cilium — and runs the same commands against the same Service on both. All output shown is from real machines.
All tests run on kind v0.27.0, node image kindest/node:v1.32.2, Cilium 1.17.3, cilium-cli v0.20.0, kernel 6.8. Both clusters are single-node; multi-node-only behaviors are out of scope. ClusterIPs change each time a cluster is recreated, so IPs may differ between sections.
The reproduction repo is at https://github.com/shinagawa-web/kubernetes-cilium-kube-proxy-replacement-debug-guide
Where forwarding rules live, and when they get written
kube-proxy
Rules were registered in netfilter
A Service's ClusterIP doesn't exist on any machine. When a Pod sends a packet to it, the destination gets rewritten to a real Pod IP somewhere in the forwarding path. kube-proxy was what registered those rewrite rules in netfilter. netfilter is the Linux kernel's packet-processing framework; iptables is the command for reading and writing its rules.
That's why iptables-save worked. The forwarding rule — which Pod a ClusterIP packet goes to — was stored as a text rule registered in the kernel, so dumping everything was enough.
Here's what the packet path looks like when a Pod connects to a ClusterIP:
client Pod
| sends packet destined for 10.96.111.7:80 (ClusterIP)
v
network stack
|
v
netfilter nat table
KUBE-SERVICES → KUBE-SVC-xxxx → KUBE-SEP-xxxx
| rewrites destination to 10.244.0.6:80 (Pod IP) here
v
Pod
tcpdump could capture the pre-DNAT packet
In this path, there's a stretch before the rewrite where the packet still carries the ClusterIP as its destination. Running tcpdump on the node with the ClusterIP as the filter caught those packets. Capturing before and after the rewrite showed exactly which Pod IP the destination changed to.
iptables-save for the rules, tcpdump for the live packets. Those two tools covered Service forwarding.
Cilium
Cluster setup
To enable kube-proxy replacement, you disable kube-proxy at cluster creation and install Cilium as the CNI. The test repo does this with kind. If you already have Cilium running, skip ahead to "No kube-proxy means no chains."
First, the kind cluster config disables both kube-proxy and the default CNI:
# cluster/kind-cilium.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
image: kindest/node:v1.32.2
networking:
disableDefaultCNI: true
kubeProxyMode: none
kubeProxyMode: none keeps kube-proxy off the nodes. disableDefaultCNI: true removes kindnet so Cilium can take the CNI slot.
After creating the cluster, install Cilium via Helm. kubeProxyReplacement=true starts Cilium in the mode where it owns Service forwarding as well.
helm install cilium cilium/cilium \
--version 1.17.3 \
--namespace kube-system \
--set kubeProxyReplacement=true \
--set k8sServiceHost="${API_SERVER_IP}" \
--set k8sServicePort="${API_SERVER_PORT}"
k8sServiceHost and k8sServicePort are the address Cilium uses to reach the Kubernetes API server. Without kube-proxy, you have to supply these explicitly. The full setup script is in scripts/setup-cilium.sh.
No kube-proxy means no chains
With this setup, kube-proxy doesn't exist. No process registers rules, so KUBE-SERVICES is never created. That's why the initial grep returned 0 lines.
The forwarding table moves to eBPF maps
The destination mapping moves to eBPF. eBPF lets you load programs into the kernel and run them at predefined hook points — on packet receive, on system call entry — without modifying the kernel itself or rebooting. The attached program can read and modify the data passing through that point.
eBPF programs are stateless across invocations — local variables don't persist between calls. State lives in maps instead: a key-value store the kernel maintains. Programs read and write there. Cilium writes the ClusterIP-to-Pod-IP mapping into these maps; on each connection, the attached program looks up the map to decide where to forward.
The rewrite happens at connect()
If only the storage location changed, you'd just learn a new command. But there's one more difference: the rewrite happens earlier.
The same connection in a Cilium environment no longer goes through netfilter at all, and the rewrite point shifts:
client Pod
| calls connect() with 10.96.89.117:80 (ClusterIP) as the destination
v
eBPF program attached to connect()
| rewrites destination to 10.0.0.183:80 (Pod IP)
v
Pod
Cilium attaches an eBPF program to connect(), the system call where an application tells the kernel what it wants to connect to. At this point, not a single byte of data has been sent. The destination address gets replaced with the Pod IP right here.
The rewrite is done before any packet is constructed, so the resulting packet carries the Pod IP as its destination — no ClusterIP packet is ever created. That's why tcpdump on a network interface never shows the ClusterIP.
This applies to connections from within the cluster. Packets arriving from outside the cluster on a NodePort are handled differently, as covered in a later section.
Two kinds of "no output"
At this point there are two separate "nothing shows up" situations with different causes. iptables-save returning 0 lines is because the forwarding rules moved from iptables to eBPF maps. tcpdump returning 0 packets when filtering on a ClusterIP is because there are no packets with that ClusterIP as a destination in that path. The causes are different, so the replacement commands are different. The next section handles the first one.
Reading the forwarding table
The test environment has a single ClusterIP Service named demo, backed by a Deployment with 2 replicas. Here's how to read the forwarding table for that Service in each environment.
kube-proxy
Take another look at those 13 lines from iptables-save | grep KUBE-SERVICES.
:KUBE-SERVICES - [0:0]
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
:KUBE-SERVICES - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:metrics cluster IP" -m tcp --dport 9153 -j KUBE-SVC-JD5MR3NA4I4DYORP
-A KUBE-SERVICES -d 10.96.111.7/32 -p tcp -m comment --comment "default/demo cluster IP" -m tcp --dport 80 -j KUBE-SVC-73JNOU5FOFXIWZLE
-A KUBE-SERVICES -d 10.96.182.1/32 -p tcp -m comment --comment "default/demo-nodeport cluster IP" -m tcp --dport 80 -j KUBE-SVC-IECF2FXKD7A6IGZV
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
The first 6 lines are chain definitions and hook points — no forwarding destination information. Lines 7 onward list each Service's entry. -d is the ClusterIP; -m comment --comment contains default/<ServiceName> cluster IP. Find the Service you want by that comment. demo is line 8 — "default/demo cluster IP" — and its ClusterIP is 10.96.111.7.
The -j KUBE-SVC-73JNOU5FOFXIWZLE at the end of that line is the jump to the backend-selection chain. Grep for that chain name to get the backend rules:
iptables-save -t nat | grep KUBE-SVC-73JNOU5FOFXIWZLE
-A KUBE-SVC-73JNOU5FOFXIWZLE ! -s 10.244.0.0/16 -d 10.96.111.7/32 -p tcp -m comment --comment "default/demo cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.6:80" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-4BM33MODP4BKIE3G
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.7:80" -j KUBE-SEP-C7KHQYF2QRUY23U4
The KUBE-SVC chain selects the backend. The first line — ! -s 10.244.0.0/16 — handles masquerading for traffic coming from outside the Pod CIDR; it's not part of backend selection. Lines 2 and 3 are the actual selection: the comment shows default/demo -> <Pod IP>:80 and --probability sets the distribution ratio. The -j KUBE-SEP-xxx at the end is the jump to the chain that executes the DNAT:
iptables-save -t nat | grep -E "KUBE-SEP-4BM33MODP4BKIE3G|KUBE-SEP-C7KHQYF2QRUY23U4"
-A KUBE-SEP-4BM33MODP4BKIE3G -p tcp -m comment --comment "default/demo" -m tcp -j DNAT --to-destination 10.244.0.6:80
-A KUBE-SEP-C7KHQYF2QRUY23U4 -p tcp -m comment --comment "default/demo" -m tcp -j DNAT --to-destination 10.244.0.7:80
--to-destination is the backend Pod IP and port.
Summary of how to read the forwarding table in a kube-proxy environment:
- ClusterIP: the
-A KUBE-SERVICES -dline (confirm the Service name in the comment) - Backend Pod IP:
--to-destinationin the corresponding KUBE-SEP chain - Weight:
--probabilityin the KUBE-SVC chain
Cilium
First, get the ClusterIP from kubectl get svc:
kubectl get svc demo
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
demo ClusterIP 10.96.24.124 <none> 80/TCP 4s
ClusterIP is 10.96.24.124. cilium-dbg service list -o json shows the frontend (ClusterIP), Service name, and backend Pod IPs all at once:
kubectl -n kube-system exec ds/cilium -- cilium-dbg service list -o json \
| jq -r '.[] | [
(.spec.id | tostring),
(.spec["frontend-address"].ip + ":" + (.spec["frontend-address"].port|tostring)),
.spec.flags.type,
(.spec.flags.namespace + "/" + .spec.flags.name),
(.spec["backend-addresses"] // [] | map(.ip + ":" + (.port|tostring)) | join(" "))
] | @tsv' | column -t
1 10.96.0.1:443 ClusterIP default/kubernetes 172.18.0.2:6443
5 10.96.0.10:53 ClusterIP kube-system/kube-dns 10.0.0.88:53 10.0.0.192:53
12 10.96.24.124:80 ClusterIP default/demo 10.0.0.40:80 10.0.0.8:80
2 0.0.0.0:30080 NodePort default/demo-nodeport 10.0.0.113:80 10.0.0.195:80
...
The 10.96.24.124 from kubectl get svc matches ID 12. The last column shows the backends: 10.0.0.40:80 and 10.0.0.8:80. The three-hop walk through KUBE-SERVICES → KUBE-SVC → KUBE-SEP in kube-proxy fits in a single line here.
In a multi-node cluster, kubectl -n kube-system exec ds/cilium execs into whichever Pod in the DaemonSet gets scheduled. In a multi-node cluster you only see the local state of that agent. The same applies to hubble observe flows — traffic on other nodes won't appear. If you know which node your target Pod is on, check kubectl get pod -n kube-system -o wide | grep cilium for that node's cilium Pod name and exec into it explicitly with kubectl -n kube-system exec <PodName>.
Packet tracing
Comparing how far tcpdump gets you when chasing ClusterIP traffic in each environment.
kube-proxy
Running tcpdump on the node and filtering on the ClusterIP captured packets. DNAT happened in netfilter, so there was a stretch before the rewrite where packets actually carried the ClusterIP as their destination. That "bracket the DNAT with two tcpdump captures" technique only worked because of that structure.
kubectl debug node/<node> -it --image=nicolaka/netshoot -- tcpdump -n -i any "host 10.96.104.4"
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
11:38:19.980617 vethc1bb8f9e In IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [S], seq 3530419686, win 64240, options [mss 1460,sackOK,TS val 4270529724 ecr 0,nop,wscale 10], length 0
11:38:19.980665 vethc1bb8f9e Out IP 10.96.104.4.80 > 10.244.0.5.54462: Flags [S.], seq 2901899611, ack 3530419687, win 65160, options [mss 1460,sackOK,TS val 4170938121 ecr 4270529724,nop,wscale 10], length 0
11:38:19.980675 vethc1bb8f9e In IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [.], ack 1, win 63, options [nop,nop,TS val 4270529724 ecr 4170938121], length 0
11:38:19.980726 vethc1bb8f9e In IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [P.], seq 1:76, ack 1, win 63, options [nop,nop,TS val 4270529724 ecr 4170938121], length 75: HTTP: GET / HTTP/1.1
11:38:19.980741 vethc1bb8f9e Out IP 10.96.104.4.80 > 10.244.0.5.54462: Flags [.], ack 76, win 64, options [nop,nop,TS val 4170938121 ecr 4270529724], length 0
11:38:19.983523 vethc1bb8f9e Out IP 10.96.104.4.80 > 10.244.0.5.54462: Flags [P.], seq 1:1545, ack 76, win 64, options [nop,nop,TS val 4170938124 ecr 4270529724], length 1544: HTTP: HTTP/1.1 200 OK
11:38:19.983538 vethc1bb8f9e In IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [.], ack 1545, win 67, options [nop,nop,TS val 4170938124 ecr 4270529724], length 0
11:38:19.983641 vethc1bb8f9e In IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [.], seq 76, ack 1545, win 67, options [nop,nop,TS val 4270529727 ecr 4170938124], length 0
11:38:19.983883 vethc1bb8f9e Out IP 10.96.104.4.80 > 10.244.0.5.54462: Flags [F.], seq 1545, ack 77, win 64, options [nop,nop,TS val 4170938124 ecr 4270529727], length 0
11:38:19.983898 vethc1bb8f9e In IP 10.244.0.5.54462 > 10.96.104.4.80: Flags [.], ack 1546, win 67, options [nop,nop,TS val 4270529727 ecr 4170938124], length 0
10 packets captured
10 packets received by filter
0 packets dropped by kernel
Packets to the ClusterIP (10.96.104.4) are captured. SYN through FIN, all there. DNAT happens in netfilter after this, so the packets with the ClusterIP destination exist in this stretch.
conntrack -L also shows the DNAT in a single entry:
kubectl debug node/<node> -it --image=nicolaka/netshoot -- conntrack -L | grep "10.96.104.4"
tcp 6 118 TIME_WAIT src=10.244.0.5 dst=10.96.104.4 sport=54462 dport=80 src=10.244.0.6 dst=10.244.0.5 sport=80 dport=54462 [ASSURED] mark=0 use=1
dst=10.96.104.4 is the ClusterIP (outbound direction); src=10.244.0.6 is the backend Pod IP (return direction). Before and after the DNAT, on one line.
Cilium
Same experiment on the Cilium cluster. Run tcpdump while curling ClusterIP 10.96.89.117 from the client Pod:
kubectl debug node/<node> -it --image=nicolaka/netshoot -- tcpdump -n -i any "host 10.96.89.117"
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
0 packets captured
0 packets received by filter
0 packets dropped by kernel
The curl succeeds. Connectivity is there. The empty capture is not a filter mistake — there are simply no packets with that ClusterIP as their destination.
connect() rewrites the destination to a Pod IP before any packet is created, so whatever hits the network already has the Pod IP. The ClusterIP never reaches the network layer.
conntrack -L filtered on the ClusterIP also returns nothing. The conntrack table contains only Pod-IP entries, but since no packets with the ClusterIP pass through netfilter, there's nothing to match.
kubectl debug node/<node> -it --image=nicolaka/netshoot -- conntrack -L | grep "10.96.89.117"
No output.
Seeing the ClusterIP translation with Hubble
tcpdump can't show the ClusterIP. So how do you trace ClusterIP traffic in a Cilium environment? Hubble is Cilium's observability component — it surfaces eBPF events as flow records. The test repo's scripts/setup-cilium.sh installs with Hubble Relay enabled. Curl the ClusterIP from the client Pod and then run hubble observe:
kubectl -n kube-system exec ds/cilium -- hubble observe --last 30 | grep -E "client|demo"
--last 30 outputs the 30 most recent flows.
Sep 7 07:23:31.106: default/client (ID:52940) <> 10.96.89.117:80 (world) pre-xlate-fwd TRACED (TCP)
Sep 7 07:23:31.106: default/client (ID:52940) <> default/demo-864f5f87b9-rvr95:80 (ID:10947) post-xlate-fwd TRANSLATED (TCP)
Sep 7 07:23:31.106: default/client:52852 (ID:52940) -> default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: SYN)
Sep 7 07:23:31.106: default/client:52852 (ID:52940) <- default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: SYN, ACK)
Sep 7 07:23:31.106: default/client:52852 (ID:52940) -> default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK)
Sep 7 07:23:31.106: default/client:52852 (ID:52940) -> default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Sep 7 07:23:31.110: default/client:52852 (ID:52940) <- default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK, PSH)
Sep 7 07:23:31.110: default/client:52852 (ID:52940) -> default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Sep 7 07:23:31.110: default/client:52852 (ID:52940) <- default/demo-864f5f87b9-rvr95:80 (ID:10947) to-endpoint FORWARDED (TCP Flags: ACK, FIN)
Focus on the first two lines. pre-xlate-fwd TRACED is the pre-translation flow, destination still recorded as 10.96.89.117:80 — the ClusterIP from kubectl get svc demo. post-xlate-fwd TRANSLATED is after, destination now the actual Pod demo-864f5f87b9-rvr95:80.
This confirms why tcpdump couldn't see the ClusterIP: the translation is done before the packet reaches the network. Hubble records both sides of the translation as eBPF events, so you can read "which ClusterIP became which Pod" directly.
NodePort is visible to tcpdump
Try NodePort on the same cluster and the result is different. Send a curl from outside the cluster to the NodePort (172.18.0.2:30080) while running tcpdump for that port on the node:
kubectl debug node/<node> -it --image=nicolaka/netshoot -- tcpdump -n -i any "port 30080"
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
07:23:24.736193 eth0 In IP 172.18.0.1.52306 > 172.18.0.2.30080: Flags [S], seq 3175489113, win 64240, options [mss 1460,sackOK,TS val 194961069 ecr 0,nop,wscale 10], length 0
07:23:24.736306 eth0 Out IP 172.18.0.2.30080 > 172.18.0.1.52306: Flags [S.], seq 2271513508, ack 3175489114, win 64308, options [mss 1410,sackOK,TS val 855246694 ecr 194961069,nop,wscale 10], length 0
07:23:24.736333 eth0 In IP 172.18.0.1.52306 > 172.18.0.2.30080: Flags [.], ack 1, win 63, options [nop,nop,TS val 194961069 ecr 855246694], length 0
07:23:24.736421 eth0 In IP 172.18.0.1.52306 > 172.18.0.2.30080: Flags [P.], seq 1:80, ack 1, win 63, options [nop,nop,TS val 194961069 ecr 855246694], length 79
...
10 packets captured
10 packets received by filter
0 packets dropped by kernel
10 packets captured. The difference is the path. NodePort packets arrive from outside the cluster as real packets, so they're not subject to socket LB. Processing happens at the tc layer (TCX), after the NIC, which is why tcpdump sees them.
cilium-dbg status --verbose shows which layers are active:
kubectl -n kube-system exec ds/cilium -c cilium-agent -- \
cilium-dbg status --verbose | grep -E "Attach Mode|Socket LB|XDP Acceleration"
Attach Mode: TCX
Socket LB: Enabled
XDP Acceleration: Disabled
Socket LB: Enabled means Cilium rewrites destinations at the socket layer (the connect() hook). ClusterIP traffic gets translated to a Pod IP there, before any packet is created — which is why tcpdump never sees it.
XDP Acceleration: Disabled means NIC-driver-level (XDP) processing is off. XDP requires a compatible NIC, so it's disabled in kind by default. Instead, as Attach Mode: TCX shows, BPF programs are attached at the tc (Traffic Control) layer. NodePort traffic is real packets from outside the cluster, not subject to socket LB, so it gets processed at that tc layer and passes through the NIC — making it visible to tcpdump.
Packet tracing summary
| kube-proxy | Cilium | |
|---|---|---|
| tcpdump on ClusterIP | captures (packet exists before DNAT) | nothing (translated at connect()) |
| conntrack on ClusterIP | captures (DNAT before/after on one line) | nothing (doesn't go through netfilter) |
| tcpdump on NodePort | captures | captures (processed at tc layer) |
| translation visibility | tcpdump before/after DNAT | Hubble pre-xlate-fwd / post-xlate-fwd |
E2E debugging flow
Starting from "Service is unreachable," walk through the investigation steps for a Cilium environment. By the end of this section, you'll have a clear path from symptom to cause.
The test repo's manifests/broken/ has two failure patterns: svc-wrong-selector.yaml (a Service with a selector that matches no Pods) and netpol-deny.yaml (a NetworkPolicy that blocks the client). The full debug script is in tests/debug-cilium.sh.
Case A — selector mismatch
Apply manifests/broken/svc-wrong-selector.yaml. The demo-broken Service has selector app: demo-v2, but every Pod in the cluster has the label app: demo. Nothing matches.
Check the ClusterIP first:
kubectl get svc demo-broken
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
demo-broken ClusterIP 10.96.171.11 <none> 80/TCP 4s
Send a curl from the client Pod:
kubectl exec -n default client -- \
curl -s --max-time 5 -o /dev/null -w '%{http_code} (%{time_total}s)' http://10.96.171.11/
000 (0.000063s)
%{http_code} returning 000 means no HTTP response. %{time_total} shows it returned immediately. When a Service has zero backends, Cilium's eBPF program rejects the connection right away — same speed as kube-proxy's REJECT --reject-with icmp-port-unreachable.
Step 1 — check Endpoints
Same command as before Cilium:
kubectl get endpoints demo-broken
NAME ENDPOINTS AGE
demo-broken <none> 8s
<none> means no Pods are registered as backends. Compare against a working Service:
kubectl get endpoints demo
NAME ENDPOINTS AGE
demo 10.0.0.117:80,10.0.0.119:80 34s
kubectl get endpoints works exactly the same as in a kube-proxy environment (from 1.33, kubectl get endpointslices is preferred).
Step 2 — check Cilium's internal state
The JSON output includes the Service name:
kubectl -n kube-system exec ds/cilium -- cilium-dbg service list -o json \
| jq -r '.[] | [
(.spec.id | tostring),
(.spec["frontend-address"].ip + ":" + (.spec["frontend-address"].port|tostring)),
.spec.flags.type,
(.spec.flags.namespace + "/" + .spec.flags.name),
(.spec["backend-addresses"] | length | tostring)
] | @tsv' | column -t
12 10.96.171.11:80 ClusterIP default/demo-broken 0
8 10.96.249.121:80 ClusterIP default/demo 2
The last column is backend count. demo (ID 8) has 2; demo-broken (ID 12) has 0. A backend count of 0 indicates endpoint shortage. Cilium registers the Service but has nowhere to forward connections.
In a kube-proxy environment, the same situation shows up in the filter table with a comment:
kubectl debug node/<node> -it --image=nicolaka/netshoot -- iptables-save -t filter | grep "10.96.171.11"
-A KUBE-SERVICES -d 10.96.171.11/32 -p tcp -m comment --comment "default/demo-broken has no endpoints" -m tcp --dport 80 -j REJECT --reject-with icmp-port-unreachable
has no endpoints is right there in the comment. Run that same command on the Cilium cluster and you get nothing. For endpoint-shortage cases (as opposed to NetworkPolicy), iptables stops explaining itself. Use cilium-dbg service list -o json | jq backend count instead. 0 means no endpoints.
Root cause
The most common reason Endpoints is empty is a selector mismatch between the Service and its Pods. Compare them directly:
kubectl get svc demo-broken -o jsonpath='{.spec.selector}'
{"app":"demo-v2"}
kubectl get pod -l app=demo -o jsonpath='{range .items[*]}{.metadata.name} {.metadata.labels}{"\n"}{end}'
demo-864f5f87b9-dccbz {"app":"demo","pod-template-hash":"864f5f87b9"}
demo-864f5f87b9-zzngq {"app":"demo","pod-template-hash":"864f5f87b9"}
The selector points to app: demo-v2; the Pods carry app: demo.
Case B — NetworkPolicy deny
Apply manifests/broken/netpol-deny.yaml. The deny-client-to-demo NetworkPolicy allows Ingress to Pods with app: demo only from Pods labeled app: not-client. The client Pod doesn't match, so all connections to demo are denied.
kubectl exec -n default client -- \
curl -s --max-time 5 -o /dev/null -w '%{http_code} (%{time_total}s)' http://10.96.249.121/
000 (5.001378s)
Same symptoms as Case A — but %{time_total} is 5 seconds. Case A (no backends) returned in 0.000063 seconds. Here, --max-time was exhausted. Packets are reaching the destination; nothing is sending a response back. Endpoints tell a different story.
Step 1 — check Endpoints
kubectl get endpoints demo
NAME ENDPOINTS AGE
demo 10.0.0.117:80,10.0.0.119:80 34s
Endpoints are healthy. That's the branch point from Case A. Empty Endpoints → check selector match and Pod readiness. Endpoints present → keep going.
Step 2 — confirm Cilium sees active backends
kubectl -n kube-system exec ds/cilium -- cilium-dbg service list -o json \
| jq -r '.[] | select(.spec["frontend-address"].ip == "10.96.249.121") | [
(.spec.id | tostring),
(.spec["frontend-address"].ip + ":" + (.spec["frontend-address"].port|tostring)),
.spec.flags.type,
(.spec.flags.namespace + "/" + .spec.flags.name),
(.spec["backend-addresses"] | length | tostring)
] | @tsv' | column -t
8 10.96.249.121:80 ClusterIP default/demo 2
Backend count is 2; Cilium knows about them. Verify the eBPF map itself agrees with what the agent reports:
kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf lb list | grep "10.96.249.121"
10.96.249.121:80/TCP (0) 0.0.0.0:0 (8) (0) [ClusterIP, non-routable]
10.96.249.121:80/TCP (1) 10.0.0.52:80/TCP (8) (1)
10.96.249.121:80/TCP (2) 10.0.0.167:80/TCP (8) (2)
service list is the agent's view of what it configured. bpf lb list reads the eBPF maps directly. They usually agree; when they diverge, the agent reports healthy backends the datapath never received. Here they match — the forwarding table is correct. Something else is dropping the packets.
Step 3 — check Hubble for packet drops
Cilium enforces NetworkPolicy directly in eBPF, so nothing shows up in iptables. Use Hubble:
kubectl exec -n default client -- curl -s --max-time 5 http://10.96.249.121/ > /dev/null
kubectl -n kube-system exec ds/cilium -- hubble observe --verdict DROPPED --last 20 | grep -E "client|demo"
Sep 9 08:07:46.753: default/client:51940 (ID:12800) <> default/demo-864f5f87b9-dccbz:80 (ID:41143) policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
Sep 9 08:07:46.753: default/client:51940 (ID:12800) <> default/demo-864f5f87b9-dccbz:80 (ID:41143) Policy denied DROPPED (TCP Flags: SYN)
Sep 9 08:07:47.777: default/client:51940 (ID:12800) <> default/demo-864f5f87b9-dccbz:80 (ID:41143) policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
Sep 9 08:07:47.777: default/client:51940 (ID:12800) <> default/demo-864f5f87b9-dccbz:80 (ID:41143) Policy denied DROPPED (TCP Flags: SYN)
Policy denied DROPPED. A NetworkPolicy is blocking the connection. policy-verdict:none INGRESS DENIED is the same verdict from a different angle. Either of these in hubble observe --verdict DROPPED points to NetworkPolicy as the cause.
Root cause
kubectl get networkpolicy deny-client-to-demo -o jsonpath='{.spec.ingress}'
[{"from":[{"podSelector":{"matchLabels":{"app":"not-client"}}}]}]
Ingress is only allowed from Pods labeled app: not-client. The client Pod doesn't qualify.
Debug flow summary
The two cases point to this sequence:
- Run
kubectl get endpoints(orkubectl get endpointslicesfrom 1.33).<none>→ check selector match and Pod readiness. - If Endpoints exist, run
cilium-dbg service list -o json | jqand check the backend count. 0 → Cilium doesn't see the backends. - If backends exist but there's no connectivity, run
hubble observe --verdict DROPPED.Policy denied→ NetworkPolicy is the cause.
Tool mapping from kube-proxy to Cilium:
| Old tool (kube-proxy) | New tool (Cilium) | What it shows |
|---|---|---|
| `iptables-save -t nat \ | grep KUBE-SERVICES` | `cilium-dbg service list -o json \ |
| {% raw %}`iptables-save -t filter \ | grep ` | backend count from `cilium-dbg service list -o json \ |
{% raw %}conntrack -L
|
cilium-dbg bpf ct list global |
Connection tracking table |
tcpdump |
hubble observe |
Per-flow packet visibility (see "Packet tracing") |
| (no equivalent) | hubble observe --verdict DROPPED |
NetworkPolicy denials |
Wrap-up
iptables-save returning 0 lines was not a broken cluster. kube-proxy doesn't exist, so nothing writes forwarding rules there — they moved to eBPF maps.
To read the forwarding table, use cilium-dbg service list -o json | jq. One command gets you the ClusterIP, Service name, and backend Pod IPs. The three-hop KUBE-SERVICES → KUBE-SVC → KUBE-SEP walk collapses to one line.
Parts of the troubleshooting workflow carry over unchanged. kubectl get endpoints works fine in a Cilium environment. After that, check cilium-dbg service list -o json | jq for backend count, and if backends are present but connectivity still fails, check hubble observe --verdict DROPPED for NetworkPolicy denials. The has no endpoints comment that kube-proxy wrote in the filter table is gone, but a backend count of 0 means the same thing.
Appendix — iptables evaluation cost grows with endpoint count
Measured while scaling demo replicas from 1 to 16. The kube-proxy column shows the number of KUBE-SVC chain rules; the Cilium column shows LB map backend slots.
replicas kube-proxy SVC rules Cilium backend slots
1 2 1
2 3 2
4 5 4
8 9 8
16 17 16
Both grow linearly with endpoint count. The difference is lookup cost. The table shows equal growth in entry count; the asymmetry is in how those entries are evaluated per new connection.
kube-proxy evaluates KUBE-SVC chain rules top to bottom to pick a backend. Each rule's --probability check either hits or falls through to the next. When the last endpoint is selected, every rule before it was evaluated first. With 16 replicas, up to 16 evaluations happen. This only applies to the first packet of a new connection — subsequent packets are handled by conntrack and skip the rules entirely. The impact shows up in environments with high new-connection rates and many Services.
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.10:80" -m statistic --mode random --probability 0.06250000000 -j KUBE-SEP-...
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.11:80" -m statistic --mode random --probability 0.06666666688 -j KUBE-SEP-...
...
-A KUBE-SVC-73JNOU5FOFXIWZLE -m comment --comment "default/demo -> 10.244.0.9:80" -j KUBE-SEP-...
Cilium computes random() % N to get a slot number and does one map lookup to find the backend. Slot 0 is the service master entry; slots 1+ are backends. With 16 Pods, it's still one lookup. This slot format is cilium-dbg bpf lb list output — it reads the eBPF maps directly, not the agent's in-memory state.
kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf lb list | grep "10.96.210.178"
10.96.210.178:80/TCP (0) 0.0.0.0:0 (8) (0) [ClusterIP, non-routable]
10.96.210.178:80/TCP (1) 10.0.0.207:80/TCP (8) (1)
10.96.210.178:80/TCP (2) 10.0.0.48:80/TCP (8) (2)
...
10.96.210.178:80/TCP (16) 10.0.0.187:80/TCP (8) (16)
