Sometimes you just want to clean up a bit and accidentally tear down half the house. That's what happened to me last week with my home lab. I wanted to free up some space on the Lenovo M920q and ran a routine Docker cleanup: delete a few unused images, remove orphaned volumes.
But when I opened my Gitea instance afterward, the page was blank. No repositories, no users, everything reset to factory defaults. All my code was gone.
The fatal error in the compose file
The answer was in my docker-compose.yml, where I had made a bad architectural mistake during setup. My volume mapping looked like this:
services:
gitea:
image: gitea/gitea:latest
container_name: gitea
volumes:
- /var/lib/docker/volumes/infra_volume/_data/gitea_data:/data
At first glance it looks like a normal path. The problem is that /var/lib/docker/volumes/ is Docker's internal storage. I had reached around Docker and pointed a bind mount straight at the raw data of another volume (infra_volume). Docker didn't see that dependency, so the cleanup job classified the original volume as unused and deleted it.
Surgical recovery from the Proxmox backup
Restoring the entire VM from backup would have taken forever and overwritten the rest of my setup. I only needed one directory out of the backup, so I had to extract the data directly from the Proxmox backup without ever booting the VM.
The backup lay as a compressed file on my storage:
vzdump-qemu-100-2026_02_23-01_00_02.vma.zst
First, I had to remove the compression:
zstd -d vzdump-qemu-100-2026_02_23-01_00_02.vma.zst
Then I extracted the raw VMA archive using Proxmox's own tool into a temporary folder:
vma extract vzdump-qemu-100-2026_02_23-01_00_02.vma /mnt/restore
Now I had a raw image file called tmp-disk-drive-scsi0.raw in front of me. You can't just mount such an image. It contains its own partitions and, in Ubuntu's case, LVM structures. I had to mount the image as a loop device:
losetup -f
losetup /dev/loop0 /mnt/restore/tmp-disk-drive-scsi0.raw
kpartx -av /dev/loop0
With the kpartx command, I made the partitions in the image visible to the host system. Since Ubuntu uses LVM, I had to wake up the volume group inside the backup:
vgscan
vgchange -ay
The system immediately found the volume group ubuntu-vg. Now I could finally mount the partition like a normal disk and navigate to my lost data:
mount -t ext4 /dev/ubuntu-vg/ubuntu-lv /mnt/vm
cd /mnt/vm/var/lib/docker/volumes/infra_volume/_data/gitea_data
There they were again. My repositories and the Gitea database were intact.
The NFS trap and the SQLite repair
Copying the data back brought the next hurdle. My new destination was an NFS share on my Synology NAS. NFS uses root_squash by default, so the root user of the Proxmox server can't push files with full permissions to the share, and a normal copy failed.
The solution was rsync without taking over permissions:
rsync -rlt --no-perms --no-owner --no-group /mnt/vm/var/... /mnt/gitea/
The relief was short-lived. The file gitea.db was corrupted, probably because the backup ran at the exact moment the database was being written to. SQLite has a repair mode for this:
sqlite3 gitea.db ".recover" | sqlite3 gitea_fixed.db
If that aborts, you can force SQLite to ignore errors during import:
sqlite3 gitea.db ".recover" | sqlite3 -init <(echo ".bail off") gitea_fixed.db
Two things I'd do differently next time. I ran .recover directly on the file sitting on the NFS mount, and SQLite's locking doesn't get along well with NFS, so that's a second corruption waiting to happen. The safer order is: copy the file to local disk, check how bad it actually is, repair it there, then rsync the fixed copy back.
sqlite3 gitea.db "PRAGMA integrity_check;"
integrity_check tells you upfront whether you're looking at a handful of misplaced pages or a completely shredded file, before you commit to a recovery pass.
The new setup
After replacing the database and cleanly unmounting everything (umount, vgchange -an, losetup -d), I rebuilt the setup to avoid the mistake.
The new compose file keeps the data on the NAS via a direct mount point instead of a path inside Docker's internal storage. Whatever Docker does on the Lenovo, the data stays on the NAS:
version: "3.9"
services:
gitea:
image: gitea/gitea:latest
container_name: gitea
restart: always
networks: [cloudflare]
ports:
- "3000:3000"
- "2222:2222"
volumes:
- /mnt/gitea:/data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:3000/api/healthz"]
interval: 30s
timeout: 5s
retries: 3
runner:
image: gitea/act_runner:latest
container_name: gitea-runner
restart: always
depends_on: [gitea]
networks: [cloudflare]
environment:
GITEA_RUNNER_NAME: CI-Runner
CONFIG_FILE: /data/config.yaml
GITEA_INSTANCE_URL: http://gitea:3000
GITEA_RUNNER_REGISTRATION_TOKEN_FILE: /run/secrets/gitea_runner_token
volumes:
- runner_data:/data
- /var/run/docker.sock:/var/run/docker.sock
networks:
cloudflare:
external: true
volumes:
runner_data:
external: true
secrets:
gitea_runner_token:
file: /data/compose/secrets/gitea/gitea_runner_token
What I took away from it
Never use /var/lib/docker/volumes/ or any of Docker's internal paths as the source for a bind mount. Use --no-perms --no-owner --no-group when rsyncing onto an NFS mount. Learn SQLite's .recover mode before you need it, but run integrity_check first and repair on local disk, not over NFS. Add a healthcheck so the next failure shows up as a failing container instead of an empty page. And a backup is only worth something once you have actually pulled a single directory out of one and put it back.
This post was originally published on www.slashgordon.link.
