Fix: kubectl exec fails with 'connection refused' — 5 real causes

Trying to exec into a running pod and Kubernetes refuses:

error: unable to upgrade connection: pod does not exist

Or:

Error from server: error dialing backend: dial tcp 10.42.0.7:10250: connect: connection refused

The pod shows Running in kubectl get pods, yet exec doesn't work. Here are the real causes and how to fix each.

Diagnose first

Before guessing, gather facts:

For RBAC-related issues, also check what permissions the current user has:

# Confirm pod is actually running
kubectl get pod <pod-name> -n <namespace>

# Check node the pod is on
kubectl get pod <pod-name> -n <namespace> -o wide

# Check the node's status
kubectl get node <node-name>

# Check kubelet logs on the node (if you have SSH access)
journalctl -u kubelet -n 100

The exact error message narrows down which of the 5 causes below applies.

Cause 1 — kubelet not reachable from API server

kubectl exec requires the API server to open a tunnel to the kubelet on port 10250. If a firewall or security group blocks that port, exec fails while everything else works.

Symptom: error like dial tcp <node-ip>:10250: connect: connection refused or i/o timeout.

Fix:

• Cloud clusters (EKS/GKE/AKS): check that the control plane security group can reach worker nodes on 10250/TCP. On EKS, this is the eksClusterSecurityGroup inbound rule on the node security group.
• Self-hosted: verify iptables or ufw allows 10250 from control plane nodes.

# On the node, verify kubelet is listening
sudo ss -tlnp | grep 10250

# From the control plane, test connectivity
nc -zv <node-internal-ip> 10250

If the port is closed, add the firewall rule. If open but still failing, jump to cause 2.

Cause 2 — kubelet is running but crashed the streaming server

kubelet has a separate HTTPS server for exec/attach/logs. It sometimes crashes without taking down the whole kubelet.

Symptom: logs works but exec and attach fail. Or intermittent failures.

Fix — restart kubelet:

sudo systemctl restart kubelet
sudo journalctl -u kubelet -f

Watch for errors during startup. If kubelet keeps crashing, the underlying issue is often disk pressure, memory pressure, or a stale container runtime socket.

Cause 3 — pod is Running but container isn't ready for TTY

Distroless containers, scratch images, or containers without a shell installed have no /bin/sh — exec fails immediately.

Symptom: OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

Fix — use an ephemeral debug container (Kubernetes 1.23+):

kubectl debug -it <pod-name> -n <namespace> \
  --image=busybox:1.36 \
  --target=<container-name>

This attaches a debug container to the same pod's namespace, giving you a shell that CAN inspect the target container's processes, network, and mounts. Works even for distroless.

Cause 4 — RBAC blocking your user from exec

You can list pods but not exec into them.

Symptom: Error from server (Forbidden): pods "<name>" is forbidden: User "<you>" cannot create resource "pods/exec"

Diagnose:

kubectl auth can-i create pods/exec -n <namespace>

If it returns no, your Role or ClusterRole is missing the pods/exec sub-resource.

Fix — grant exec permission:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-exec
  namespace: YOUR_NAMESPACE
rules:
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]

Bind it to your user with a RoleBinding and retry.

Cause 5 — kubectl version mismatch with cluster

Very old kubectl talking to a much newer cluster (or vice versa) can hit exec-protocol incompatibilities.

Symptom: exec starts, prints a partial banner, then disconnects. Or error: unable to upgrade connection: unauthorized.

Diagnose:

kubectl version

The default output shows both client and server versions. Kubernetes officially supports client/server skew of +/- 1 minor version. Beyond that, exec is unreliable.

Fix — align kubectl version:

# macOS
brew install kubernetes-cli
# or pinned to a specific minor version:
brew install kubernetes-cli@1.29

# Linux — pinned install
curl -LO "https://dl.k8s.io/release/v1.29.0/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Match kubectl to your cluster's minor version and retry.

The universal exec debug flow

Every stuck-exec case, run in order:

# 1. Pod actually running?
kubectl get pod <pod> -n <ns>

# 2. Node healthy?
kubectl get node -o wide

# 3. Can I even see logs? (rules out kubelet-total-failure)
kubectl logs <pod> -n <ns> --tail=5

# 4. RBAC permission for exec?
kubectl auth can-i create pods/exec -n <ns>

# 5. Try debug container (bypasses distroless issue)
kubectl debug -it <pod> --image=busybox --target=<container>

Ninety percent of exec failures resolve after step 5.

Prevention

Bake these into every cluster setup:

• Explicit security group rules for control-plane to node on 10250/TCP; documented in Terraform, not vibes-based
• Ephemeral containers enabled on all managed clusters (default in K8s 1.25+; verify with kubectl api-resources | grep ephemeral)
• Standardized debug image in a private registry — a common debug:latest with curl, bind-utils, netcat, iputils pre-installed
• RBAC role called debugger with pods/exec permission, granted temporarily via kubectl auth reconcile for incident response
• Regular kubectl version audits across the team — pin to cluster version in your dotfiles

Bottom line

kubectl exec failing usually points at one of five layers: network to kubelet, kubelet streaming server, missing shell in the container, RBAC, or version skew. Diagnose with kubectl get pod -o wide, kubectl auth can-i create pods/exec, and kubectl debug. The debug container is a strong Swiss army knife — reach for it early rather than fighting distroless exec issues directly.