A payments integration that took 40 milliseconds outside the cluster took 180 inside it, consistently, with the same code and the same network path. The vendor's latency numbers were fine. Our application spans showed the time being spent before the TLS handshake started, in a region of the trace we had never instrumented.
It was DNS, and it was our own doing. Kubernetes writes a resolv.conf into every pod with a search list of namespace, service, and cluster suffixes, plus options ndots:5. That last number means any name containing fewer than five dots is treated as relative and tried against every search domain first. api.payments-vendor.com has two dots. So each lookup went to api.payments-vendor.com.prod.svc.cluster.local, then .svc.cluster.local, then .cluster.local, then the node's search domain, and only on the fifth attempt as an absolute name. Four guaranteed NXDOMAINs before the one that works, doubled because the resolver asked for A and AAAA records, and every one of them a round trip to CoreDNS.
Under load this stopped being merely wasteful. CoreDNS was serving several times the query volume it needed to, its cache hit rate was poor because most of what it saw were misses that do not cache usefully, and during a traffic spike we started seeing five second stalls, which is the classic musl and glibc resolver timeout showing up as "the vendor is slow".
The fixes were unglamorous. Services that call external endpoints got a dnsConfig with ndots: 2, which puts most public hostnames back on the fast path. Our HTTP client configuration now uses fully qualified names with a trailing dot for third-party hosts, so the search list is skipped entirely. We enabled NodeLocal DNSCache, which keeps a cache on each node and takes the conntrack pressure off the cluster DNS pods. And CoreDNS request rate, cache hit ratio and NXDOMAIN share went onto a dashboard, because none of that had ever been visible.
Latency inside a cluster is rarely just the network. It is the defaults that were chosen to make service discovery convenient, applied to names that were never going to be services.
– Sergey Shinder
