I renamed a live Proxmox node and twelve guests never noticed

The host had the wrong name. I had given it the good name, the one I wanted for the whole cluster, back when it was the only machine I owned. Then I bought two more machines, and the good name was sitting on the box least likely to survive the next two years.

So the node needed a new name, and it needed it while twelve production guests were running on it. Nextcloud, the password vault, the reverse proxy, the DNS resolvers, the monitoring stack. Things my family notices when they stop.

Renaming a Linux box is one command. Renaming a Proxmox node is not, because Proxmox keeps a per-node directory of state keyed by hostname, inside a filesystem you do not fully control.

The problem

Proxmox stores cluster configuration in /etc/pve, which looks like a normal directory and is not one. It is a FUSE mount backed by pmxcfs, a small replicated database that syncs across every node in the cluster. Under it, /etc/pve/nodes/<hostname>/ holds everything that belongs to that specific machine: the guest configs in qemu-server/ and lxc/, the per-node firewall rules, the SSL certificate.

Change the hostname and that directory path no longer matches the machine. The guest configs are still on disk, but they are sitting under a name the node no longer answers to. Proxmox will happily boot into that state and show you an empty node with no VMs on it, which is a memorable thing to see at half past midnight.

The instinct is to move the directory. That instinct is wrong, and it is wrong in a way that fails safe the first time and unsafe the second.

What I tried, and what failed

My first move was the obvious one:

mv /etc/pve/nodes/oldname /etc/pve/nodes/newname

mv: cannot move '/etc/pve/nodes/oldname' to '/etc/pve/nodes/newname': Directory not empty

The target already existed. The moment pmxcfs saw the new hostname, it created the new node directory itself, complete with empty qemu-server/ and lxc/ subdirectories. You are not moving into empty space. The daemon got there first.

Fine, so copy instead:

cp -av /etc/pve/nodes/oldname/qemu-server/. /etc/pve/nodes/newname/qemu-server/

This is where it got interesting. Run that too early, in the seconds before pmxcfs has created the subdirectories, and it does not error in a way that stops you. It reports nothing copied and returns. If you are working through a checklist at speed, "no output" reads exactly like "done."

I then did the thing I am going to write a whole separate post about, which is that I deleted the source directory before confirming the copy had landed. Twelve guest configs, gone from the only canonical copy. I will spare you the details here; the short version is that a pre-rename backup saved me and production never actually went down.

The reason production survived is worth understanding, because it is the mechanism that makes this whole operation safe. A running LXC or QEMU guest does not read /etc/pve continuously. It reads its config once, at start, and then runs from what it already holds. pve-manager re-reads the directory on a cold start. So for as long as nothing restarts, the configs can be missing entirely and every workload keeps serving. That is a real forgiveness window, and it is the only reason a rename on a live node is a reasonable thing to attempt.

It is forgiveness, not permission. A power cut during that window and I would have had twelve running services and no idea how any of them were configured.

The fix

Here is the sequence that works, with the verification gates that stop the failure modes above. Take a backup of the whole /etc/pve tree to somewhere outside /etc/pve first. Not to a snapshot, to a plain directory on local disk you can read with cat when everything else is broken.

cp -a /etc/pve /root/etc-pve.backup-pre-rename-$(date +%Y%m%d-%H%M%S)

Then:

# 1. Set the new name in both places
sed -i 's/oldname/newname/' /etc/hostname
# keep the old name as an alias in /etc/hosts through the transition
hostnamectl set-hostname newname

# 2. WAIT. Let pmxcfs notice and build the new directory itself.
#    Do not mv. Do not mkdir. Just watch for it to appear.
until [ -d /etc/pve/nodes/newname/qemu-server ]; do sleep 1; done
ls -la /etc/pve/nodes/newname/

# 3. Copy the state across
cp -av /etc/pve/nodes/oldname/qemu-server/. /etc/pve/nodes/newname/qemu-server/
cp -av /etc/pve/nodes/oldname/lxc/.         /etc/pve/nodes/newname/lxc/
cp -av /etc/pve/nodes/oldname/host.fw       /etc/pve/nodes/newname/ 2>/dev/null

# 4. VERIFY THE COUNTS MATCH before you delete anything
ls /etc/pve/nodes/oldname/qemu-server/ | wc -l
ls /etc/pve/nodes/newname/qemu-server/ | wc -l
ls /etc/pve/nodes/oldname/lxc/ | wc -l
ls /etc/pve/nodes/newname/lxc/ | wc -l

# 5. Only when those numbers agree, and honestly you can just leave the old
#    directory in place forever, it costs a few kilobytes
rm -rf /etc/pve/nodes/oldname/

# 6. Restart the services that cache the node identity
systemctl restart pve-cluster
systemctl restart pvedaemon pveproxy pvestatd

# 7. Regenerate the web UI certificate under the new name
pvecm updatecerts -f

# 8. The step everyone misses
pve-firewall restart

That last line is the one that cost me the second half of the night. Proxmox marks the firewall as having pending changes after a hostname change and does not apply them. The running kernel rules still scope to the old node name, so the host can reach outward while nothing can reach in. It is a silent, asymmetric loss of connectivity and it looks nothing like a firewall problem. That one has its own write-up on Thursday.

Verify before you call it done:

pve-firewall status     # must NOT say "pending changes"
pvecm status            # quorum intact, node listed under the new name
qm list && pct list     # every guest present and still running

And ping the node from another machine on the network. Not from itself.

Why this works

The whole procedure comes down to one idea: pmxcfs owns that directory tree, and you are a guest in it. Every step that fights the daemon fails, and every step that waits for the daemon works.

mv fails because the daemon has already created the destination. mkdir is unnecessary for the same reason. The copy has to wait because the subdirectories appear a beat after the parent does, and a copy into a path that does not exist yet is not an error, it is just nothing. The verification gate exists because "nothing happened" and "everything happened" produce the same silence on the terminal.

The service restarts are ordered deliberately. pve-cluster owns pmxcfs itself, so it goes first and the rest of the stack picks up the new identity behind it. pvecm updatecerts -f matters because the web UI certificate carries the old hostname in its subject, and browsers do care.

What I'd do differently

I would skip step 5 entirely. Nothing about a rename requires deleting the old node directory, and it is the only command in the whole procedure that can lose you something. Leaving it costs a few kilobytes and buys you a second copy of every guest config for nothing.

I would also treat the gate in step 4 as the actual work rather than as paperwork. Every other step here either succeeds visibly or fails loudly at you. Steps 3 and 4 are the only two that can quietly do nothing, and they sit directly in front of the one command you cannot undo.

The rename took about thirty minutes. Cleaning up after my own impatience took another fifteen, and that part gets its own write-up next week. Every guest kept running throughout, which says more about how Proxmox treats a live config directory than it does about my procedure.

A note on the addresses

Hostnames and addresses in this post are documentation placeholders. Any IPs elsewhere in this series use the ranges reserved for documentation under RFC 5737 and RFC 2606. The architecture is real; the specific names are stand-ins so nothing here maps to a live target.