Introduction
If you are learning Kubernetes, you will eventually hear a simple but important statement:
etcd is the database of a Kubernetes cluster.
That description is short, but it has a major consequence: if the etcd data is lost or corrupted, the Kubernetes control plane can lose the information it needs to know what exists in the cluster.
In this lab, I practiced an end-to-end etcd backup and restore workflow on a single-node kubeadm cluster running on a machine named osboxes.
The exercise covered:
- identifying the etcd pod and confirming that the control plane was healthy;
- locating the static etcd manifest;
- inspecting the etcd configuration;
- installing
etcdctlandetcdutl; - creating an etcd snapshot;
- validating the snapshot;
- deliberately creating an
nginxDeployment so there was something meaningful to restore; - deleting the Deployment;
- restoring the etcd snapshot into a separate data directory;
- changing the static etcd manifest to use the restored data;
- troubleshooting an etcd pod that remained
Pending; - restarting kubelet;
- confirming that etcd eventually returned to
Running.
The important part of this exercise was not simply getting a successful backup command. It was understanding what each error meant and how to reason from the evidence in the terminal.
The terminal session showed the etcd pod running before the backup, with the control plane components also healthy. Later, after changing the etcd static pod configuration, etcd became Pending; the investigation eventually led to restarting kubelet, after which etcd became 1/1 Running. The original terminal output is the source for the commands and results reproduced throughout this article.
1. First: What is etcd?
Before touching commands, it helps to understand what we are backing up.
Kubernetes has many components, but the important relationship for this exercise is roughly:
kubectl
|
v
kube-apiserver
|
v
etcd
|
+---- Kubernetes cluster state
The API server is the main interface through which Kubernetes clients interact with the cluster. etcd is the persistent key-value store used by Kubernetes to store cluster state.
That state includes information such as:
- Kubernetes objects;
- Deployments;
- Pods and their specifications;
- Services;
- ConfigMaps;
- Secrets;
- cluster configuration and metadata.
A useful mental model for a beginner is:
The Kubernetes API is how you ask Kubernetes about the cluster; etcd is where much of that cluster state is persisted.
This is why backing up etcd is fundamentally different from simply exporting Kubernetes YAML.
2. Verify that the cluster is healthy first
Before taking a backup, I first checked the cluster.
Terminal
┌──(osboxes㉿osboxes)-[~]
└─$ k get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-74c68c8864-fjtnv 0/1 ContainerCreating 0 3m10s
kube-system calico-node-dzlcp 0/1 Init:2/3 0 3m10s
kube-system coredns-589f44dc88-l522w 0/1 ContainerCreating 0 8m56s
kube-system coredns-589f44dc88-nzsh9 0/1 ContainerCreating 0 8m56s
kube-system etcd-osboxes 1/1 Running 0 9m6s
kube-system kube-apiserver-osboxes 1/1 Running 0 9m9s
kube-system kube-controller-manager-osboxes 1/1 Running 0 9m5s
kube-system kube-proxy-d8krd 1/1 Running 0 8m56s
kube-system kube-scheduler-osboxes 1/1 Running 0 9m8s
The most important observation here is:
etcd-osboxes 1/1 Running
The API server, controller manager, scheduler and kube-proxy were also running.
Some networking components were still starting, so I watched the pods:
┌──(osboxes㉿osboxes)-[~]
└─$ k get pods -A -w
Eventually CoreDNS and Calico became ready.
I then checked the node:
┌──(osboxes㉿osboxes)-[~]
└─$ k get nodes
NAME STATUS ROLES AGE VERSION
osboxes Ready control-plane 10m v1.36.3
At this point the single control-plane node was Ready.
The initial cluster state is important because a backup should be taken from a functioning etcd instance rather than blindly assuming that etcd is healthy.
3. An important distinction: Kubernetes YAML backup vs etcd backup
Before working directly with etcd, I also ran:
┌──(osboxes㉿osboxes)-[~]
└─$ k get all -A -o yaml > backup.yaml
This produced:
┌──(osboxes㉿osboxes)-[~]
└─$ ls
backup.yaml Documents kubectl Pictures Public Videos
Desktop Downloads Music Projects Templates VM-Share
This is useful, but it is not the same thing as an etcd snapshot.
The command:
k get all -A -o yaml > backup.yaml
exports Kubernetes resources returned by kubectl get all.
An etcd snapshot, on the other hand, is a snapshot of the etcd backend itself.
For an etcd disaster-recovery exercise, the etcd snapshot is the important artifact.
4. Finding the etcd static pod
This cluster was created with kubeadm, so etcd was running as a static pod.
I initially made a typo:
┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kkubenetes/manifests
bash: cd: /etc/kkubenetes/manifests: No such file or directory
The problem was simply the spelling:
kkubenetes
instead of:
kubernetes
I then made another typo:
┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kubenetes/manifests
bash: cd: /etc/kubenetes/manifests: No such file or directory
Again, the correct directory is:
/etc/kubernetes/manifests
Eventually:
┌──(osboxes㉿osboxes)-[~]
└─$ cd /etc/kubernetes/manifests
and:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls
etcd.yaml
kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yaml
This directory is extremely important on a kubeadm control-plane node.
The files in /etc/kubernetes/manifests are static pod manifests. The kubelet watches this directory and makes sure the described control-plane containers are running.
That means editing etcd.yaml is effectively changing how the control-plane etcd process is launched.
5. Another wrong assumption: trying to use Docker
I also tried:
┌──(osboxes㉿osboxes)-[~]
└─$ docker exec -it osboxes bash
Error response from daemon: No such container: osboxes
This failed because the Kubernetes node name osboxes is not necessarily a Docker container name.
In this setup, the node was running Kubernetes with containerd, and the etcd pod later showed a container ID beginning with:
containerd://
So the important lesson is:
Do not assume that a Kubernetes node or Pod name is a Docker container name.
If you need to inspect a Kubernetes container, first determine which container runtime the node is using.
6. Inspecting the etcd manifest
The first attempt to read the manifest without elevated privileges also failed:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ cat etcd.yaml
cat: etcd.yaml: Permission denied
The solution was:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo cat etcd.yaml
The manifest revealed several important values.
For example:
- --advertise-client-urls=https://10.0.2.15:2379
- --cert-file=/etc/kubernetes/pki/etcd/server.crt
- --client-cert-auth=true
- --data-dir=/var/lib/etcd
- --key-file=/etc/kubernetes/pki/etcd/server.key
- --listen-client-urls=https://127.0.0.1:2379,https://10.0.2.15:2379
It also showed that the etcd data directory was originally:
/var/lib/etcd
and the TLS certificates were under:
/etc/kubernetes/pki/etcd/
This information is essential because etcdctl needs to connect to the correct endpoint and authenticate with the appropriate certificates.
7. Install etcdctl
The machine did not initially have the etcd client installed, so I installed it:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo apt install etcd-client
The package installed successfully:
Installing:
etcd-client
Summary:
Upgrading: 0, Installing: 1, Removing: 0, Not Upgrading: 205
I then checked the snapshot command:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ETCDCTL_API=3 etcdctl snapshot
The output showed:
COMMANDS:
restore Restores an etcd member snapshot to an etcd directory
save Stores an etcd node backend snapshot to a given file
status [deprecated] Gets backend snapshot status of a given file
The key command for the backup is:
etcdctl snapshot save
8. ETCDCTL_API=3
I also explicitly exported:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ export ETCDCTL_API=3
For the lab, this made it explicit that etcdctl should use the v3 API.
Then:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl snapshot
again showed the snapshot subcommands.
For a beginner, the important idea is:
ETCDCTL_API=3
selects the etcd API version used by the client.
9. Creating the first etcd snapshot — and making a typo
I attempted the backup with:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl ---endpoint=https://127.0.0.1:2379,https://10.0.2.15:2379 \
> --cacert=/etc/kubernetes/pki/etcd/ca.crt \
> --cert=/etc/kubernetes/pki/etcd/server.crt \
> --key=/etc/kubernetes/pki/etcd/server.key \
> snapshot save /opt/etcd-backup.db
The error was:
Error: bad flag syntax: ---endpoint=https://127.0.0.1:2379,https://10.0.2.15:2379
The mistake was the number of hyphens.
I typed:
---endpoint
instead of:
--endpoints
There were actually two problems worth noticing:
- There were three hyphens.
- The correct flag is
--endpoints, plural.
This is a classic CLI problem: the command can look almost correct to a human while being completely invalid to the parser.
10. The next error: permission denied
I corrected the flag and tried:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db
The command reached the certificate files but failed with:
Error: open /etc/kubernetes/pki/etcd/server.key: permission denied
This made sense.
The etcd private key is protected, and the regular user did not have permission to read it.
The solution was to run the command with sudo:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db
This time the backup succeeded.
11. The successful etcd backup
The successful command produced:
{"level":"info","ts":"2026-08-13T12:11:59.649233-0400","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/opt/etcd-backup.db.part"}
{"level":"info","ts":"2026-08-13T12:11:59.660384-0400","logger":"client","caller":"v3/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2026-08-13T12:11:59.660467-0400","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2026-08-13T12:11:59.831074-0400","logger":"client","caller":"v3/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2026-08-13T12:11:59.894020-0400","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"3.9 MB","took":"now"}
{"level":"info","ts":"2026-08-13T12:11:59.894158-0400","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/opt/etcd-backup.db"}
Snapshot saved at /opt/etcd-backup.db
The resulting backup file was:
/opt/etcd-backup.db
The snapshot was approximately:
3.9 MB
This was the first major milestone.
12. Validating the snapshot
I initially tried:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdctl --write-out=table snapshot status snapshot.db
The client responded:
Deprecated: Use `etcdutl snapshot status` instead.
Error: stat snapshot.db: no such file or directory
There were two lessons here.
Lesson 1: The filename was wrong
The backup was saved as:
/opt/etcd-backup.db
not:
snapshot.db
Lesson 2: The command was deprecated
The output explicitly recommended:
etcdutl snapshot status
So I attempted:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status snapshot.db
but etcdutl was not installed.
The system suggested:
sudo apt install etcd-server
I installed the package.
After installation, I still used the wrong filename once:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status snapshot.db
Error: stat snapshot.db: no such file or directory
Then I used the correct path:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --write-out=table snapshot status /opt/etcd-backup.db
This produced:
Error: open /opt/etcd-backup.db: permission denied
Again, the issue was file permissions.
So I used sudo:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db
The result was:
┌──────────┬──────────┬────────────┬────────────┐
│ HASH │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ a633855e │ 5973 │ 1298 │ 3.9 MB │
└──────────┴──────────┴────────────┴────────────┘
This confirmed that the snapshot was readable and contained etcd data.
13. Why create an nginx Deployment?
A backup is much more meaningful if we have something we can verify after restoring it.
At this point, I created an nginx Deployment.
My first attempt was:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy nginx
error: required flag(s) "image" not set
The command was missing the container image.
I then tried:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy --image=nginx
error: exactly one NAME is required, got 0
See 'kubectl create deployment -h' for help and examples
Now the image was present, but the Deployment name was missing.
The correct command was:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k create deploy nginx --image=nginx
deployment.apps/nginx created
This is a good beginner lesson:
kubectl create deployment <NAME> --image=<IMAGE>
So:
kubectl create deployment nginx --image=nginx
means:
- create a Deployment;
- call it
nginx; - use the
nginxcontainer image.
14. Waiting for the nginx Pod
Immediately after creation:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME READY STATUS RESTARTS AGE
pod/nginx-7f8fbb96d-9f7wm 0/1 ContainerCreating 0 4s
The Pod remained in:
ContainerCreating
I watched it:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -w
After waiting, I checked again.
The Pod eventually became:
nginx-7f8fbb96d-9f7wm 1/1 Running 0 3m16s
The describe output explained why it initially took time:
Events:
Type Reason Age From Message
---- ------ --- ---- -------
Normal Scheduled 83s default-scheduler Successfully assigned default/nginx-7f8fbb96d-9f7wm to osboxes
Normal Pulling 82s kubelet spec.containers{nginx}: Pulling image "nginx"
The Pod had been scheduled successfully. The kubelet was simply pulling the image.
This is an important troubleshooting principle:
ContainerCreatingdoes not automatically mean something is broken. Check the Pod events before changing anything.
Eventually:
k get po
NAME READY STATUS RESTARTS AGE
nginx-7f8fbb96d-9f7wm 1/1 Running 0 3m16s
15. Take another backup after the nginx Deployment exists
This step is important because the first snapshot was taken before the nginx Deployment was created.
After nginx was running, I created another snapshot:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db
The snapshot was again:
3.9 MB
and the command ended with:
Snapshot saved at /opt/etcd-backup.db
I validated it:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db
Result:
┌──────────┬──────────┬────────────┬────────────┐
│ HASH │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ eb688bf4 │ 6934 │ 918 │ 3.9 MB │
└──────────┴──────────┴────────────┴────────────┘
Notice that the revision changed.
The backup now represented a later state of the cluster.
16. Verify the state we want to recover
At this point:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME READY STATUS RESTARTS AGE
pod/nginx-7f8fbb96d-9f7wm 1/1 Running 0 4m1s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 76m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/nginx 1/1 1 1 4m2s
NAME DESIRED CURRENT READY AGE
replicaset.apps/nginx-7f8fbb96d 1 1 1 4m2s
The important object is:
deployment.apps/nginx
It was healthy.
Now the test was simple:
If I delete nginx, can I use the etcd snapshot to recover the cluster state that existed when the snapshot was taken?
17. Delete nginx
I deleted the Deployment:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k delete deploy nginx
deployment.apps "nginx" deleted from default namespace
Then:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get all
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 76m
The nginx Deployment was gone.
This gave us the desired recovery scenario:
BEFORE BACKUP
|
+-- nginx Deployment exists
|
v
TAKE ETCD SNAPSHOT
|
v
DELETE nginx
|
v
nginx no longer exists
|
v
RESTORE SNAPSHOT
|
v
expect previous cluster state
18. First restore mistake: wrong snapshot filename
I attempted:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ etcdutl --data-dir /opt/etcd/backup snapshot restore snapshot.db
The restore process immediately told me:
Error: open snapshot.db: no such file or directory
Again, the filename was wrong.
The actual snapshot was:
/opt/etcd-backup.db
This is a very common command-line mistake: creating a file with one name and later trying to restore a similarly named file.
The first restore attempt was therefore not an etcd failure. It was simply a path/filename error.
19. Restore the snapshot into a separate data directory
Instead of overwriting the active etcd data directory immediately, I restored the snapshot into a separate directory:
/var/lib/etcd-restore-from-backup
The command used was:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot restore /opt/etcd-backup.db \
--data-dir=/var/lib/etcd-restore-from-backup
The output included:
Deprecated: Use `etcdutl snapshot restore` instead.
and then:
restoring snapshot
followed by:
Trimming membership information from the backend...
and:
added member
Finally:
restored snapshot
This was the actual successful restore of the snapshot into the new data directory.
20. Why use a separate restore directory?
The original etcd data directory was:
/var/lib/etcd
The restored copy was:
/var/lib/etcd-restore-from-backup
This distinction is important.
It allows the restored data to be prepared without immediately destroying the existing data directory.
For a beginner, think of it like restoring a database backup into a new folder first:
Original:
/var/lib/etcd
Restored:
/var/lib/etcd-restore-from-backup
Only after the restored data exists do we configure etcd to use it.
21. More path mistakes while inspecting the restored data
I attempted:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ cd /var/lib/data
bash: cd: /var/lib/data: No such file or directory
Again, the directory did not exist.
The restore directory was:
/var/lib/etcd-restore-from-backup
I also tried:
sudo cd /var/lib/etcd-restore-from-backup
and got:
sudo: cd: command not found
sudo: "cd" is a shell built-in command, it cannot be run directly.
This error is worth understanding.
cd is a shell built-in command. sudo normally executes an external command, so:
sudo cd ...
does not work.
Instead, because the directory required elevated permissions, I used:
sudo ls /var/lib/etcd-restore-from-backup
which showed:
member
That confirmed that the restore had created the expected etcd member directory structure.
22. Editing the static etcd manifest
The next step was to configure the static etcd pod to use the restored data directory.
The original manifest used:
--data-dir=/var/lib/etcd
The modified configuration used:
--data-dir=/var/lib/etcd-restore-from-backup
The corresponding hostPath was also changed to:
/var/lib/etcd-restore-from-backup
The resulting Pod description later confirmed:
--data-dir=/var/lib/etcd-restore-from-backup
and:
Path: /var/lib/etcd-restore-from-backup
This is how the restored database was connected to the etcd process.
23. Why moving the manifest files caused disruption
During the troubleshooting process, I temporarily moved the static pod manifests out of the manifest directory:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo mv * /tmp
Then:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls -lrt
total 0
The directory was empty.
I then restored the manifests:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo mv /tmp/*.yaml .
and confirmed:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ ls -lrt
total 16
-rw------- 1 root root 3939 Aug 13 11:07 kube-apiserver.yaml
-rw------- 1 root root 3230 Aug 13 11:07 kube-controller-manager.yaml
-rw------- 1 root root 1726 Aug 13 11:07 kube-scheduler.yaml
-rw------- 1 root root 2596 Aug 13 12:37 etcd.yaml
This demonstrates how sensitive the static pod manifest directory is.
When the kubelet stops seeing a static pod manifest, the corresponding static pod can disappear.
For this reason, moving files in /etc/kubernetes/manifests should be done carefully.
24. Another small typo
I accidentally ran:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get oi
error: the server doesn't have a resource type "oi"
The intended command was clearly not oi.
The correct command for Pods was:
k get po
This is a harmless error, but it illustrates another important habit:
When
kubectlreports that a resource type does not exist, first check the command spelling before assuming the Kubernetes API is broken.
25. The RBAC error after the manifest changes
After the static pod manipulation, this happened:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po
Error from server (Forbidden): pods is forbidden: User "kubernetes-admin" cannot list resource "pods" in API group "" in the namespace "default"
This was different from the earlier errors.
This time, the Kubernetes API server was responding, but the authenticated user was being denied access.
The command:
sudo k get po
was also not a solution:
sudo: k: command not found
The reason is that k was a shell alias, and aliases normally are not available to sudo in that way.
More importantly, sudo does not automatically solve Kubernetes RBAC problems.
I then checked the system namespace explicitly:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -n=kube-system
This returned the control-plane Pods.
That helped separate the problem from the actual etcd state.
26. etcd was now Pending
The output showed:
etcd-osboxes 0/1 Pending
I first tried:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po etcd-osboxes
Error from server (NotFound): pods "etcd-osboxes" not found
Again, the namespace mattered.
The correct command was:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po -n=kube-system etcd-osboxes
The description showed the important configuration:
Status: Pending
but also:
Image: registry.k8s.io/etcd:3.6.8-0
and:
--data-dir=/var/lib/etcd-restore-from-backup
The Pod also showed the restored directory mounted:
/var/lib/etcd-restore-from-backup from etcd-data
and:
Path: /var/lib/etcd-restore-from-backup
The kubelet events showed:
Normal Pulled ...
Normal Created ...
Normal Started ...
So the container had actually been created and started even though the Kubernetes status remained Pending.
This is an important troubleshooting clue:
Do not rely on one status field alone. Read the full Pod description and events.
27. Trying to read etcd logs
I tried:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k logs etcd-osboxes -n kube-system
Error from server (BadRequest): container "etcd" in pod "etcd-osboxes" is not available
I also accidentally ran:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k logs
which returned:
error: expected 'logs [-f] [-p] (POD | TYPE/NAME) [-c CONTAINER]'.
POD or TYPE/NAME is a required argument for the logs command
The second error was simply because kubectl logs requires a Pod or workload name.
The first error was more interesting: Kubernetes did not consider the container available for normal log retrieval.
I then checked the Pod in wide format:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get pod etcd-osboxes -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
etcd-osboxes 0/1 Pending 0 4m58s <none> osboxes <none> <none>
The lack of a Pod IP was another indication that the Pod was not fully ready.
28. Inspecting kubelet
Since etcd was a static Pod, the kubelet was a key component to inspect.
I ran:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo systemctl status kubelet --no-pager
The result showed:
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; disabled; preset: disabled)
Active: active (running)
The kubelet was running.
The logs also showed repeated activity around the time of the problem.
At this point, rather than making more changes to the etcd manifest, I restarted kubelet:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ sudo systemctl restart kubelet
Then:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ systemctl daemon-reload
29. The recovery
After restarting kubelet, I checked the system namespace again:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k get po -n=kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-74c68c8864-fjtnv 1/1 Running 5 (7m47s ago) 95m
calico-node-dzlcp 1/1 Running 0 95m
coredns-589f44dc88-l522w 1/1 Running 0 101m
coredns-589f44dc88-nzsh9 1/1 Running 0 101m
etcd-osboxes 1/1 Running 0 6m51s
kube-apiserver-osboxes 1/1 Running 0 101m
kube-controller-manager-osboxes 1/1 Running 1 (7m32s ago) 101m
kube-proxy-d8krd 1/1 Running 0 101m
kube-scheduler-osboxes 1/1 Running 1 101m
This was the desired state.
Most importantly:
etcd-osboxes 1/1 Running
30. Confirming the restored etcd configuration
I then described the etcd Pod again:
┌──(osboxes㉿osboxes)-[/etc/kubernetes/manifests]
└─$ k describe po -n=kube-system etcd-osboxes
The important portion was:
Status: Running
IP: 10.0.2.15
The etcd container was:
State: Running
Ready: True
Restart Count: 0
And the command still showed:
--data-dir=/var/lib/etcd-restore-from-backup
The mounted volume was:
/var/lib/etcd-restore-from-backup from etcd-data
with the host path:
/var/lib/etcd-restore-from-backup
This confirmed that etcd was now running using the restored data directory.
31. What happened to the nginx Deployment?
The exercise was designed around the snapshot that was taken while nginx existed.
The important sequence was:
1. Create nginx Deployment
2. Wait for nginx to become Running
3. Take etcd snapshot
4. Delete nginx
5. Restore the snapshot
6. Point etcd at restored data
7. Restart kubelet
8. Confirm etcd is Running
The backup validation showed a snapshot revision of:
6934
with:
918 total keys
3.9 MB total size
The restore operation successfully rebuilt an etcd data directory.
The final terminal state confirmed that etcd was successfully running from:
/var/lib/etcd-restore-from-backup
The terminal session provided here does not contain a final k get all after the successful etcd recovery showing nginx reappearing, so I would not claim that nginx restoration was independently verified from the terminal evidence.
That distinction matters.
The exercise successfully demonstrated:
- snapshot creation;
- snapshot validation;
- snapshot restoration;
- switching the static etcd Pod to restored data;
- recovery of the etcd Pod.
The source terminal output does not provide a final explicit post-recovery k get all proving that the nginx Deployment was recreated from the restored snapshot.
32. The complete clean backup command
For this environment, the working backup command was:
sudo etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db
The important pieces are:
Endpoint
--endpoints=https://127.0.0.1:2379
This tells etcdctl where etcd is listening for client connections.
CA certificate
--cacert=/etc/kubernetes/pki/etcd/ca.crt
This allows the client to verify the etcd server certificate.
Client certificate
--cert=/etc/kubernetes/pki/etcd/server.crt
This identifies the client to the TLS-enabled etcd server.
Private key
--key=/etc/kubernetes/pki/etcd/server.key
This goes with the client certificate.
Snapshot command
snapshot save /opt/etcd-backup.db
This tells etcdctl to create a backend snapshot at the specified path.
33. The clean snapshot validation command
The session showed that etcdctl snapshot status is deprecated and recommended etcdutl.
The working validation command was:
sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db
Example output from the session:
┌──────────┬──────────┬────────────┬────────────┐
│ HASH │ REVISION │ TOTAL KEYS │ TOTAL SIZE │
├──────────┼──────────┼────────────┼────────────┤
│ eb688bf4 │ 6934 │ 918 │ 3.9 MB │
└──────────┴──────────┴────────────┴────────────┘
This is useful because a backup file existing on disk does not, by itself, prove that it is a valid etcd snapshot.
34. The clean restore command
The restore performed successfully with:
sudo etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot restore /opt/etcd-backup.db \
--data-dir=/var/lib/etcd-restore-from-backup
The terminal explicitly reported:
Deprecated: Use `etcdutl snapshot restore` instead.
So for a newer workflow, prefer the modern etcdutl snapshot restore command rather than copying the deprecated command blindly.
35. Important troubleshooting lessons
Error 1: Wrong directory
cd /etc/kkubenetes/manifests
and:
cd /etc/kubenetes/manifests
failed because of spelling mistakes.
Fix
cd /etc/kubernetes/manifests
Error 2: Docker container not found
docker exec -it osboxes bash
returned:
No such container: osboxes
Lesson
A Kubernetes node name is not automatically a Docker container name.
Error 3: Permission denied reading etcd files
cat etcd.yaml
returned:
Permission denied
Fix
sudo cat etcd.yaml
The same principle applied to the protected etcd private key.
Error 4: Invalid etcdctl flag
---endpoint=...
returned:
Error: bad flag syntax
Fix
Use the correctly spelled flag:
--endpoints=...
Error 5: Private key permission denied
Error: open /etc/kubernetes/pki/etcd/server.key: permission denied
Fix
Run the etcd client command with sufficient privileges:
sudo etcdctl ...
Error 6: Wrong snapshot filename
snapshot.db
did not exist.
The actual file was:
/opt/etcd-backup.db
Lesson
Always verify the exact backup path before restoring.
Error 7: etcdctl snapshot status deprecated
The client reported:
Deprecated: Use `etcdutl snapshot status` instead.
Fix
Install/use etcdutl and run:
sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db
Error 8: etcdutl not installed
The command:
etcdutl
was initially unavailable.
Fix
The system suggested installing etcd-server, which provided the utility in this environment.
Error 9: sudo cd
This failed:
sudo cd /var/lib/etcd-restore-from-backup
with:
sudo: cd: command not found
Lesson
cd is a shell built-in.
Use commands such as:
sudo ls /var/lib/etcd-restore-from-backup
or start a privileged shell if you genuinely need one.
Error 10: Pod name not found
This:
k describe po etcd-osboxes
returned:
pods "etcd-osboxes" not found
because the command defaulted to the default namespace.
Fix
k describe po -n=kube-system etcd-osboxes
Error 11: kubectl logs without a Pod
k logs
returned:
POD or TYPE/NAME is a required argument for the logs command
Fix
Provide the resource:
k logs <pod-name>
and specify the namespace when needed.
Error 12: kubectl Forbidden
After the control-plane changes:
Error from server (Forbidden): pods is forbidden:
User "kubernetes-admin" cannot list resource "pods"
This was an authorization/RBAC problem rather than a shell permission problem.
sudo did not solve it because Kubernetes authorization is separate from Linux sudo.
Error 13: etcd remained Pending
The etcd Pod stayed:
0/1 Pending
even though the container had been created and started according to the events.
The investigation included:
k describe po -n=kube-system etcd-osboxes
and:
sudo systemctl status kubelet --no-pager
The practical recovery step that resolved the state in this lab was:
sudo systemctl restart kubelet
After that:
etcd-osboxes 1/1 Running
36. Why static Pods matter in this exercise
The etcd Pod was not created with a normal Deployment such as:
kind: Deployment
Instead, it was defined by:
/etc/kubernetes/manifests/etcd.yaml
That makes it a static Pod.
The kubelet watches the manifest directory and manages the Pod locally.
This explains several things that happened during the exercise:
- changing
etcd.yamlaffected etcd; - temporarily moving manifests out of the directory affected the control plane;
- restarting kubelet caused it to re-read and reconcile the manifests;
- etcd could be controlled without creating a normal Kubernetes Deployment.
For kubeadm clusters, understanding static Pods is essential when troubleshooting the control plane.
37. A safer mental model for etcd restore
Do not think of restore as:
"copy backup file over etcd"
A better mental model is:
ETCD SNAPSHOT
|
v
restore snapshot
|
v
new etcd data directory
|
v
configure etcd to use restored directory
|
v
kubelet recreates/restarts static etcd Pod
|
v
etcd becomes healthy
|
v
Kubernetes API becomes available against restored state
This is why the exercise used:
/var/lib/etcd-restore-from-backup
instead of immediately destroying:
/var/lib/etcd
38. What I would do differently next time
The lab worked, but it also exposed several areas where the workflow could be cleaner.
1. Check the exact paths first
Before typing a long command:
ls -l /etc/kubernetes/pki/etcd/
ls -l /opt/etcd-backup.db
This avoids guessing filenames and permissions.
2. Inspect the manifest before changing it
First:
sudo cp /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml.backup
Then edit the manifest.
That gives you a known-good copy if something goes wrong.
3. Prefer the current etcd utilities
The terminal explicitly showed that:
etcdctl snapshot status
is deprecated and recommended:
etcdutl snapshot status
Likewise, the restore output recommended:
etcdutl snapshot restore
4. Avoid unnecessary movement of all manifests
This command:
sudo mv * /tmp
is powerful and potentially disruptive.
Moving all control-plane manifests affects more than etcd.
A safer approach is to modify only the required manifest and keep backups of the original.
5. Verify each stage
After every major step:
k get pods -A
or:
k get pods -n kube-system
and inspect:
k describe pod <pod> -n kube-system
if anything looks abnormal.
39. A compact recovery checklist
For a kubeadm single-control-plane lab similar to this one:
Identify etcd
k get pods -n kube-system | grep etcd
Inspect the manifest
cd /etc/kubernetes/manifests
sudo cat etcd.yaml
Identify:
--data-dir
--listen-client-urls
--cert-file
--key-file
--trusted-ca-file
Create snapshot
sudo etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db
Validate snapshot
sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db
Restore to a separate directory
sudo etcdutl snapshot restore /opt/etcd-backup.db \
--data-dir=/var/lib/etcd-restore-from-backup
Configure etcd to use the restored directory
Update:
--data-dir=/var/lib/etcd-restore-from-backup
and the corresponding hostPath.
Restart/reconcile kubelet if required
sudo systemctl restart kubelet
Verify
k get pods -n kube-system
Look for:
etcd-osboxes 1/1 Running
40. What this lab actually taught me
The most valuable part of this exercise was not memorizing one command.
It was learning how the pieces connect:
Kubernetes
|
+-- kube-apiserver
|
+-- kube-controller-manager
|
+-- kube-scheduler
|
+-- kubelet
|
+-- static Pod manifests
|
+-- etcd
|
+-- /var/lib/etcd
The backup process interacted directly with etcd:
etcd
|
+-- TLS certificates
|
+-- etcdctl
|
+-- snapshot
|
+-- /opt/etcd-backup.db
The restore process then created:
/var/lib/etcd-restore-from-backup
and the static Pod was changed to consume that restored data.
Finally, kubelet reconciled the static Pod and etcd returned to:
1/1 Running
41. Final takeaway
An etcd backup is not just another Kubernetes YAML export.
The core workflow is:
1. Verify etcd is healthy.
2. Find the etcd endpoint and TLS certificates.
3. Take an etcd snapshot.
4. Validate the snapshot.
5. Restore the snapshot into a separate data directory.
6. Configure the etcd static Pod to use the restored data.
7. Let kubelet reconcile the static Pod.
8. Verify etcd and the Kubernetes control plane.
The terminal session also demonstrated something that documentation often hides: the commands do not always work on the first attempt.
There were:
- spelling mistakes;
- incorrect flags;
- permission problems;
- wrong filenames;
- deprecated commands;
- missing utilities;
- namespace mistakes;
- RBAC errors;
- attempts to use
sudowhere it did not apply; - a static etcd Pod that remained
Pending; - and finally a kubelet restart that brought etcd back to
Running.
Those failures were not separate from the learning experience. They were the learning experience.
The most useful troubleshooting habit from this lab is simple:
When Kubernetes gives you an error, do not immediately change random things. Read the error, identify which layer produced it, inspect the relevant component, make the smallest reasonable correction, and verify the result.
That is the difference between memorizing Kubernetes commands and actually learning how the control plane works.
Terminal command reference from the lab
Cluster health
k get pods -A
k get pods -A -w
k get nodes
k get all
k get all -A
Static Pod manifests
cd /etc/kubernetes/manifests
ls
sudo cat etcd.yaml
Install tooling
sudo apt install etcd-client
sudo apt install etcd-server
Configure etcdctl
export ETCDCTL_API=3
Backup
sudo etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save /opt/etcd-backup.db
Validate
sudo etcdutl --write-out=table snapshot status /opt/etcd-backup.db
Test workload
k create deploy nginx --image=nginx
k get po
k get po -w
k describe po nginx-7f8fbb96d-9f7wm
k get all
Delete workload
k delete deploy nginx
Restore
sudo etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot restore /opt/etcd-backup.db \
--data-dir=/var/lib/etcd-restore-from-backup
Troubleshoot etcd
k get po -n=kube-system
k describe po -n=kube-system etcd-osboxes
k get pod etcd-osboxes -n kube-system -o wide
k logs etcd-osboxes -n kube-system
sudo systemctl status kubelet --no-pager
sudo systemctl restart kubelet
Final verification
k get po -n=kube-system
k describe po -n=kube-system etcd-osboxes
