Day 51: A Rollout Only Starts When the Template Changes, and Azure Keeps Only Half Your Key

The AWS track finished yesterday, so from today the cloud half of each post comes from the Azure track instead. The series keeps its name, so everything stays in one place. Day 51 of DevOps, Day 1 of Azure.

Both tasks today turn on what gets kept. A rolling update keeps the old version around so you can go back to it. Azure keeps one half of your SSH key and hands the other half to you exactly once.

One Kubernetes task, one Azure task. Roll a Deployment to a new image without downtime, then create an SSH key pair for Azure VMs. The tasks come from the KodeKloud Engineer platform.

A rollout starts only when the template changes

kubectl describe deployment nginx-deployment
kubectl set image deployment/nginx-deployment nginx-container=nginx:1.17
kubectl rollout status deployment/nginx-deployment

The describe comes first for a reason. set image takes the container name, not the Deployment name, so nginx-container has to match the container in the pod template exactly. Get it wrong and nothing updates, because there is no such container.

Kubernetes is precise about what starts a rollout: a Deployment's rollout is triggered if and only if the pod template changes, and other updates, such as scaling, do not trigger one. Changing the image changes the template. Changing the replica count does not.

The default strategy is RollingUpdate, with two settings that both default to 25%. maxUnavailable is how far below the desired count you can drop during the update, rounded down. maxSurge is how far above it you can go, rounded up.

Those rounding directions are what make a single replica safe. With one pod, 25% rounds down to zero unavailable and up to one extra, so Kubernetes starts the new pod before removing the old one. Even a one-pod Deployment updates without a gap.

kubectl rollout status then watches until it finishes, and Kubernetes documents its exit status as 0 on success and 1 when the Deployment exceeds its progress deadline. That is what belongs in a script after an update, instead of a guessed sleep.

The old version is kept on purpose

The update does not modify the existing ReplicaSet. It makes a new one:

kubectl get replicasets

The new ReplicaSet scales up, the old one scales down to zero, and the old one stays. Kubernetes keeps ten by default, and that retained ReplicaSet is exactly what kubectl rollout undo goes back to. Tomorrow's task uses it.

There were three ways to make this change: kubectl set image, kubectl edit, or editing the manifest and running kubectl apply. All three start the same rollout. They do not leave the same thing behind. The first two change the live object, so a manifest file on disk is now out of date, and the next time someone applies that file it quietly puts the old image back. Only the apply route keeps the file as the truth.

Azure keeps only the public half

The Azure task was an SSH key pair for virtual machines.

az sshkey create --name "$KEY_NAME" --resource-group "$RG"

The resource this creates is a Microsoft.Compute/sshPublicKeys object, and its only key property is the public key. There is nowhere in it for a private key to live. Microsoft documents that each newly created key is also stored locally, and the output tells you where:

Private key is saved to "/root/.ssh/1757564000_123456".
Public key is saved to "/root/.ssh/1757564000_123456.pub".

That file is the only copy of the private key that will ever exist. No API returns it. Lose it and the key pair is useless for login, and the fix is a new pair and resetting access on every VM that used the old one.

The filename is the other trap. It is not id_rsa, so ssh will not find it by default. Every login needs -i with the full path, or the file needs renaming to something memorable straight away.

Two things in my own notes turned out to be wrong when I checked them.

I had written that Windows VMs do not accept Ed25519 keys. Microsoft's documentation lists the same two supported types for Windows as for Linux, RSA of at least 2048 bits and Ed25519. The real difference is that Azure does not provision SSH public keys to Windows machines automatically at all, of any type.

And I had uploaded a key with --public-key @~/.ssh/nautilus-key.pub. The @ tells the Azure CLI to read the value from a file, which is right. But bash only expands a tilde at the start of a word, and after @ it is not at the start, so the shell passes the tilde through untouched. Microsoft's examples use absolute paths. @$HOME/.ssh/nautilus-key.pub works because the shell expands $HOME anywhere in the word.

Coming from AWS

The one-chance rule is the same on both. AWS keeps the public key and returns the private key inside the JSON response, so forgetting to capture KeyMaterial means it scrolls past once and is gone. Azure writing it to a file is harder to lose, and the filename is easier to misplace.

One more thing worth knowing. Deleting the Azure key resource does not lock anyone out of VMs made with it. The public key was copied into each VM when it was created, and the VM has no ongoing link back to the resource. The resource is a convenience for provisioning, not a control point for access.

What gets kept, and by whom

Kubernetes keeps the old ReplicaSet so you can go back. Azure keeps the public key so it can hand it to new VMs. The private key is kept by exactly one party, you, in a file with an unhelpful name.

So here is the Day 51 question. For the credentials your systems depend on, do you know where every copy of the private half lives right now?

Day 51 down. Forty-nine to go.