I Deliberately Destroyed My Kubernetes Cluster at 2 AM. Here's What Died First.
Chaos engineering is not about breaking things. It's about discovering that your "production-grade" homelab is held together by hope and a single etcd snapshot before someone else finds out for you.
The Setup
I was lying in bed at 1:47 AM, staring at the ceiling, unable to sleep.
Not because of caffeine. Because of a thought that had been gnawing at me for weeks:
If one of my nodes died right now, would my cluster actually survive?
I run a 4-node bare-metal Kubernetes cluster on Talos Linux. Dell OptiPlex control plane. Three Raspberry Pi workers. Cilium eBPF. ArgoCD. Longhorn distributed storage. Prometheus. Grafana. The whole cloud-native stack, shoehorned into $220 of scrap hardware and stubbornness.
From the outside, it looks solid. ArgoCD syncs green. Cilium status shows healthy. Longhorn volumes are replicated across three nodes. I have etcd snapshots every 6 hours to S3. On paper, I'm resilient.
But I had never actually tested it.
Not a controlled test. Not a graceful node drain. I mean chaos. Sudden death. The kind of failure that happens at 3 AM when a power supply dies, or a kernel panics, or a neighbor's construction crew hits the wrong breaker.
So I got out of bed, walked to my desk, and installed Chaos Mesh.
Why Chaos Engineering on a Homelab?
Professionally, I design AWS infrastructure with multi-AZ failover, auto-scaling groups, and managed services that abstract failure away. At Siemens, if an EKS node dies, the managed node group replaces it before I finish reading the alert.
But my homelab has no managed control plane. No AWS SLA. No auto-repair. If a Pi's USB boot drive corrupts, that node is gone until I physically fix it.
I needed to know:
- What dies first when a worker vanishes? Not "what should die" — what actually dies.
- Does Longhorn really failover? Three replicas sound great until you realize two of them were on the same node.
- Does Cilium handle network partitions? Or does it just... stop routing?
- How long does ArgoCD stay useful? If the control plane loses the GitOps controller, can I still reason about state?
- What's my actual MTTR? Mean Time To Recovery — not theoretical, measured with a stopwatch and cold sweat.
I wasn't planning to learn. I was planning to find out.
The Weapon
I installed Chaos Mesh directly into the cluster:
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm install chaos-mesh chaos-mesh/chaos-mesh \
--namespace chaos-testing \
--create-namespace \
--set chaosDaemon.runtime=containerd \
--set chaosDaemon.socketPath=/run/containerd/containerd.sock
Chaos Mesh is a CNCF sandbox project that injects failure into Kubernetes. It can kill pods, stress CPU, corrupt networks, simulate disk failures, and even tamper with DNS. It runs as controllers inside your cluster — the failures are real, not simulated.
I spend 20 minutes reading the docs. Then I stop reading and start attacking.
Attack 1: Pod Chaos (Kill Random Pods Every 30 Seconds)
Target: All pods in the production namespace (where I run kube-radar, job-digest, MarketPulse API, and PostgreSQL)
Experiment:
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
name: random-pod-killer
spec:
action: pod-kill
mode: all
duration: "5m"
selector:
namespaces:
- production
scheduler:
cron: "@every 30s"
What I expected: Pods restart. Kubernetes self-heals. ArgoCD shows sync. Business as usual.
What actually happened:
| Time | Event |
|---|---|
| T+0s | PostgreSQL pod killed. StatefulSet recreates it in 8 seconds. |
| T+30s | MarketPulse API pod killed. Deployment recreates it in 4 seconds. No impact — I have 2 replicas. |
| T+60s | Prometheus pod killed. Data loss: the last 15 minutes of scrape data vanished because I use emptyDir for Prometheus storage. |
| T+90s | kube-radar pod killed. No impact — it's a CLI CronJob, not a long-running service. |
| T+120s |
PostgreSQL killed again. This time, the new pod schedules to rpi-03, which already has a Longhorn replica. But the active replica was on rpi-01. Longhorn takes 34 seconds to failover and attach the volume to rpi-03. During those 34 seconds, MarketPulse API returns 500 because it can't reach the database. |
| T+5m | Experiment ends. I have 6 minutes of missing Prometheus data and a database failover slower than my patience. |
Lesson 1: Kubernetes recreates pods fast. Stateful workloads don't. A StatefulSet restart plus volume re-attachment is an eternity for a dependent service.
Attack 2: Network Partition (Isolate One Worker from the Control Plane)
Target: rpi-02 — completely isolated from all other nodes
Experiment:
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
name: partition-rpi-02
spec:
action: partition
mode: all
duration: "3m"
selector:
nodes:
- rpi-02
direction: both
target:
selector:
namespaces:
- kube-system
mode: all
What I expected: rpi-02 goes NotReady. Workloads on it get rescheduled. Cilium handles the network split gracefully.
What actually happened:
| Time | Event |
|---|---|
| T+0s | |
| T+15s | Cilium on rpi-02 starts logging endpoint regeneration failure. It's trying to sync BPF maps with the Cilium operator but can't reach it. |
| T+45s | The node finally transitions to NotReady because the kubelet's node lease can't be renewed. But the 45-second grace period means pods on rpi-02 are still considered "running" by the scheduler. |
| T+60s | A Longhorn volume with replicas on rpi-02 and rpi-03 loses quorum. Longhorn marks the volume as Degraded. Read operations still work. Write operations hang. |
| T+90s | MarketPulse API (running on rpi-01) tries to write to PostgreSQL. The write hangs because the database volume is degraded. The Go HTTP server goroutine blocks. After 30 seconds, the client times out. |
| T+120s | Prometheus can't scrape metrics from pods on rpi-02 because they're network-isolated. It marks them as DOWN. My Grafana dashboard looks like a massacre. |
| T+3m | Partition removed. Cilium takes 18 seconds to rebuild BPF maps. Longhorn takes 47 seconds to re-replicate the missing replica. The cluster recovers. But the 3-minute window had 2 database write timeouts and 1 failed API request. |
Lesson 2: Network partition + StatefulSet + distributed storage = distributed deadlock. The node stays Ready long enough to break things but not long enough to migrate workloads cleanly.
Attack 3: Node Chaos (Kill a Worker Node Entirely)
Target: rpi-01 — simulate total node death (power loss)
Experiment:
# Physical simulation: pull the power cable on rpi-01
# (Yes, I literally unplugged it. This is why I test at 2 AM.)
What I expected: Node goes NotReady after 40 seconds. Pods reschedule. Longhorn promotes the other replica to primary. Service continues.
What actually happened:
| Time | Event |
|---|---|
| T+0s | Power cable pulled. rpi-01 is dead. No graceful shutdown. No disk sync. |
| T+5s | Talos on the control plane detects the node is unresponsive. But the node controller waits for the default --node-monitor-grace-period=40s before marking it NotReady. |
| T+30s | During those 30 seconds, kube-proxy (Cilium eBPF) still routes traffic to pods on rpi-01 because the endpoint slices haven't been updated yet. Requests to MarketPulse API hit a dead pod IP and time out. |
| T+40s | Node finally marked NotReady. Pod eviction starts. The eviction timeout is another 5 minutes by default. I didn't change it. I'm an idiot. |
| T+45s | Longhorn detects the node is gone. It has replicas on rpi-02 and rpi-03. It promotes the rpi-02 replica to primary. But PostgreSQL was the active workload on rpi-01 — the pod hasn't been rescheduled yet because eviction hasn't timed out. |
| T+2m | I manually delete the PostgreSQL pod on rpi-01 (it's stuck in Terminating because the node is dead). The StatefulSet creates a new pod on rpi-02. Longhorn attaches the volume. PostgreSQL starts crash recovery because the previous instance died uncleanly. |
| T+3m30s | PostgreSQL finishes WAL replay. MarketPulse API reconnects. My SIEM shows 3.5 minutes of total database unavailability. |
| T+5m | Kubernetes finally evicts all remaining pods from rpi-01 automatically. They're already rescheduled by now because I intervened manually. The automatic eviction did nothing useful. |
Lesson 3: Default Kubernetes timeouts are designed for cloud nodes that recover. For bare metal, 40 seconds + 5 minutes is an eternity. You need to tune --node-monitor-grace-period and pod-eviction-timeout to match your hardware reality.
Attack 4: Stress Chaos (CPU Burn on the Control Plane)
Target: Dell OptiPlex control plane — 100% CPU for 2 minutes
Experiment:
apiVersion: chaos-mesh.org/v1alpha1
kind: StressChaos
metadata:
name: cpu-burn-control-plane
spec:
duration: "2m"
selector:
nodes:
- optiplex-control-plane
stressors:
cpu:
workers: 4
load: 100
What I expected: Control plane slows down. API latency increases. But the cluster stays functional.
What actually happened:
| Time | Event |
|---|---|
| T+0s | CPU stress applied. 4 workers pinned to 100%. |
| T+15s |
kubectl get nodes latency jumps from 200ms to 4.2 seconds. |
| T+30s | etcd starts logging apply request took too long. The WAL fsync is competing with stress-ng for disk I/O. |
| T+60s | Cilium agent on the control plane can't reach the API server fast enough for endpoint updates. It logs list-watcher timed out. Pods on other nodes start losing network identity. |
| T+75s | ArgoCD application controller crashes with a context deadline exceeded error. It was trying to list resources and gave up. When it restarts, it re-triggers a full sync of all 45 manifests. |
| T+90s | Longhorn manager on the control plane can't renew its leader election lease. It steps down. The new leader takes 12 seconds to take over. During those 12 seconds, volume attachment requests queue up. |
| T+2m | Stress removed. CPU returns to normal. But ArgoCD is still syncing. Longhorn is still re-electing. etcd has a 200MB backlog of WAL entries to compact. It takes 90 seconds for the cluster to feel "normal" again. |
Lesson 4: The control plane is not infinitely resilient. On bare metal with no CPU limits, a single noisy process can cascade through etcd → API server → Cilium → ArgoCD → Longhorn. In the cloud, this doesn't happen because the control plane is managed and isolated. On bare metal, you are the SRE.
Attack 5: DNS Chaos (Corrupt CoreDNS Responses)
Target: CoreDNS in kube-system
Experiment:
apiVersion: chaos-mesh.org/v1alpha1
kind: DNSChaos
metadata:
name: corrupt-dns
spec:
action: error
mode: all
duration: "2m"
selector:
namespaces:
- kube-system
scope: inner
target:
mode: all
selector:
namespaces:
- production
What I expected: DNS failures. Services can't resolve hostnames. Health checks fail.
What actually happened:
| Time | Event |
|---|---|
| T+0s | DNS chaos applied. CoreDNS returns NXDOMAIN for 50% of queries. |
| T+5s | MarketPulse API tries to connect to postgresql.production.svc.cluster.local. Gets NXDOMAIN. Retries. Fails. Returns 500. |
| T+10s | job-digest Python scraper tries to resolve api.linkedin.com through cluster DNS. Gets NXDOMAIN. The urllib3 retry logic waits 3 seconds, retries, waits 6 seconds, retries, gives up. The CronJob fails. |
| T+20s | Prometheus can't resolve scrape targets. All targets show DOWN. |
| T+30s |
Cilium Hubble Relay can't resolve hubble-peer.kube-system.svc.cluster.local. It stops receiving flow data. My network observability goes blind. |
| T+60s | ArgoCD can't resolve github.com to check for manifest updates. Sync status shows Unknown instead of Synced. |
| T+2m | Chaos removed. DNS recovers in 5 seconds because CoreDNS caches are in-memory. But the failed CronJob doesn't automatically re-run. The 500 errors in MarketPulse triggered my alerting. My phone buzzed. I remembered it's 2:47 AM and my wife is sleeping. I silence the alert. |
Lesson 5: DNS is not a luxury. When DNS breaks, everything breaks simultaneously because everything assumes DNS works. In a homelab, you don't have Route 53 health checks or multi-region DNS. You have one CoreDNS replica and a dream.
The Scorecard
| Attack | Expected | Reality | MTTR |
|---|---|---|---|
| Pod Chaos | Full recovery in <10s | 34s database failover + data loss | ~45s |
| Network Partition | Graceful isolation | 3m distributed deadlock | ~3m |
| Node Death | Auto-reschedule in 40s | 3.5m manual intervention needed | ~3.5m |
| CPU Stress | Slowdown, no outage | 90s cascading recovery | ~90s |
| DNS Chaos | Service failures | Total cluster blindness | ~5s (after removal) |
Summary: My cluster survived everything. But "survived" doesn't mean "handled gracefully." Every attack revealed a gap between "Kubernetes works" and "Kubernetes works when things go wrong."
What I Fixed the Next Morning
1. Tuned Kubernetes failure detection
# Kubelet configuration
nodeStatusUpdateFrequency: 10s
nodeMonitorGracePeriod: 20s
podEvictionTimeout: 60s
On bare metal, nodes don't come back. Don't wait 5 minutes to accept reality.
2. Added a second CoreDNS replica with anti-affinity
replicas: 2
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
k8s-app: kube-dns
topologyKey: kubernetes.io/hostname
One CoreDNS death should not blind the cluster.
3. Moved Prometheus to Longhorn-backed storage
spec:
volumeClaimTemplates:
- metadata:
name: prometheus-data
spec:
storageClassName: longhorn
resources:
requests:
storage: 20Gi
emptyDir for monitoring data is a mistake. Promote it to persistent storage.
4. Configured pod disruption budgets
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: postgres-pdb
spec:
minAvailable: 1
selector:
matchLabels:
app: postgresql
This prevents voluntary evictions from destabilizing critical workloads during node drains or chaos experiments.
5. Automated the chaos experiments
I created a Kubernetes CronJob that runs a subset of these experiments every Sunday at 3 AM:
apiVersion: batch/v1
kind: CronJob
metadata:
name: weekly-chaos
spec:
schedule: "0 3 * * 0"
jobTemplate:
spec:
template:
spec:
containers:
- name: chaos
image: chaos-mesh/chaos-daemon:latest
command: ["/bin/sh", "-c"]
args:
- |
kubectl apply -f /experiments/pod-chaos.yaml
sleep 300
kubectl delete -f /experiments/pod-chaos.yaml
restartPolicy: OnFailure
If you're not breaking it on purpose, someone will break it by accident.
3 Lessons That Transfer to Production
1. Hope is not a strategy
Before this night, my disaster recovery plan was: "I have etcd snapshots and Longhorn replicas. I'll figure it out." After 5 attacks, I have specific MTTR numbers, specific failure modes, and a specific list of what to fix. In production, "I'll figure it out" is not an SRE principle.
2. Defaults are not your friends
Kubernetes defaults are tuned for cloud providers with managed control planes, fast node replacement, and redundant infrastructure. On bare metal — or even in a cost-optimized cloud environment — those defaults are dangerously optimistic. Tune them or suffer at 2 AM.
3. Observability is the only thing that separates chaos from panic
During the DNS attack, the only reason I knew what was happening was because I had Prometheus, Grafana, and Cilium Hubble. Without them, I would have been guessing. With them, I could trace the failure chain: DNS → service resolution → API timeout → 500 response. Observability doesn't prevent failure. It prevents blind failure.
TL;DR
I installed Chaos Mesh on my homelab cluster and attacked it 5 ways: random pod kills, network partitions, node death, CPU starvation, and DNS corruption.
The cluster survived everything. But gracefully? No. Database failover took 34 seconds. Network partition caused a 3-minute distributed deadlock. Node death needed manual intervention. CPU stress triggered cascading failures through etcd, Cilium, ArgoCD, and Longhorn. DNS chaos blinded the entire cluster in 30 seconds.
I fixed 5 things the next morning: tuned failure detection, added CoreDNS redundancy, moved Prometheus to persistent storage, configured PodDisruptionBudgets, and automated weekly chaos experiments.
If you run Kubernetes anywhere — cloud, bare metal, or a drawer full of Raspberry Pis — break it yourself before the universe breaks it for you.
# Start here
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm install chaos-mesh chaos-mesh/chaos-mesh \
--namespace chaos-testing --create-namespace
What's your MTTR? Have you ever actually measured it? Drop it in the comments — or admit you don't know. Both are valid.
