Stop Rebuilding the Whole OS Image: Practical systemd-sysext and confext on Linux

Immutable and image-based Linux hosts are great until you need one more tool on /usr, a debug binary that was deliberately left out of the base image, or a temporary /etc overlay for a fleet flag flip.

Rebuilding the whole OS image for that is slow. Dropping files straight onto a read-only root is either impossible or a recipe for drift. systemd-sysext and systemd-confext solve a narrower problem cleanly: merge extension images into the live hierarchy with OverlayFS, then unmerge them without rewriting the base OS.

This is not a package manager. It is not a container runtime. And it is not the same thing as portable services.

What sysext and confext actually do

Per systemd-sysext(8) and the UAPI.4 Extension Images specification:

When you merge:

1. systemd finds installed extension images
2. builds an OverlayFS stack over the host hierarchy
3. overmounts /usr + /opt (sysext) or /etc (confext)

When you unmerge, the overlay goes away and the original host tree is visible again.

Important constraints from the man page:

• Files outside the supported hierarchies in an image are ignored. A sysext with /etc/foo will not put anything into host /etc.
• Extensions are read-only by default. On a mutable host, merging them makes the overlaid base directory read-only unless you enable mutability.
• There is no dependency solver. An extension must carry what it needs (beyond what the base OS already provides).
• Extensions are supposed to be additive. OverlayFS can shadow or whiteout files, but that is discouraged.

sysext vs portable services (do not mix these up)

If you are shipping a daemon that should stay sandboxed, prefer portable services. If you need strace, a locally built libfoo, or a fleet-wide /etc toggle on an otherwise immutable image, use sysext/confext.

Image formats and search paths

Supported formats match other systemd image tooling (nspawn, RootImage=):

1. Plain directories or btrfs subvolumes
2. GPT disk images following the Discoverable Partitions Spec
3. Naked filesystem images (erofs, squashfs, ext4, …) as *.raw
4. Optional dm-verity authenticity on disk images

sysext search paths (first useful for symlinks; primary install location last in practice for bulk images):

• /etc/extensions/
• /run/extensions/
• /var/lib/extensions/ ← primary place for real images
• In the initrd: /.extra/sysext/ (populated by systemd-stub from ESP companions)

confext search paths:

• /run/confexts/
• /var/lib/confexts/ ← primary
• /usr/lib/confexts/
• /usr/local/lib/confexts/

Directories = directory-based extensions. Files ending in .raw = disk-image extensions.

There is no per-image enable bit: everything installed is eligible for merge. To mask a lower-precedence image, place an empty directory with the same name under /etc/extensions/ (sysext). Kernel cmdline knobs can disable auto-merge entirely: systemd.sysext=0, systemd.confext=0 (and rd.* variants in the initrd).

Version matching with extension-release

Every sysext needs:

/usr/lib/extension-release.d/extension-release.NAME

Every confext needs:

/etc/extension-release.d/extension-release.NAME

NAME must match the image/directory name (see os-release(5) / extension-release rules). Matching against the host os-release:

• ID= must match the host, or be _any
• If ID= is not _any:
  • SYSEXT_LEVEL= (sysext) or CONFEXT_LEVEL= (confext) must match when present
  • otherwise VERSION_ID= must match
• Optional ARCHITECTURE= must match the kernel arch unless _any
• Optional EXTENSION_RELOAD_MANAGER=1 asks for a manager reload after apply
• Do not ship /usr/lib/os-release inside a sysext — it would override host OS identity after merge

You can force a merge past version checks with --force (lab only).

Lab: directory-based sysext in 10 minutes

This lab builds a tiny additive extension that drops a script under /usr/local… wait: /usr/local is under /usr, so it works. Prefer /usr vendor paths for real images; /opt is also valid for third-party trees.

# Read host identity you must match (unless ID=_any)
grep -E '^(ID|VERSION_ID|SYSEXT_LEVEL)=' /etc/os-release

# Example: allow any OS ID for a throwaway lab image
NAME="devtools-lab"
ROOT="/var/lib/extensions/${NAME}"

sudo rm -rf "$ROOT"
sudo mkdir -p \
  "$ROOT/usr/lib/extension-release.d" \
  "$ROOT/usr/bin"

# Identity file NAME must match directory name
sudo tee "$ROOT/usr/lib/extension-release.d/extension-release.${NAME}" >/dev/null <<'EOF'
ID=_any
ARCHITECTURE=_any
# Optional self-description with SYSEXT_ prefix:
SYSEXT_ID=devtools-lab
SYSEXT_VERSION_ID=0.1.0
EOF

# Payload: one additive binary/script
sudo tee "$ROOT/usr/bin/sysext-hello" >/dev/null <<'EOF'
#!/bin/sh
echo "hello from systemd-sysext"
EOF
sudo chmod 0755 "$ROOT/usr/bin/sysext-hello"

Inspect and merge:

systemd-sysext list
systemd-sysext status

# First merge (fails if already merged — then use refresh)
sudo systemd-sysext merge
# After installing/removing images on an already-merged host:
# sudo systemd-sysext refresh

command -v sysext-hello
sysext-hello
# hello from systemd-sysext

findmnt /usr
# expect overlay on /usr when sysext is active

systemd-sysext status

Unmerge cleanly:

sudo systemd-sysext unmerge
command -v sysext-hello || echo "gone — base OS restored"

refresh caveats

refresh is unmerge + merge. The man page is explicit: there is a brief window where neither overlay is mounted, so extension files disappear momentarily even if the same extension remains installed. Plan service restarts and long-running readers accordingly.

matching a real distro instead of ID=_any

For production images built beside the base OS:

# Example pattern — values must match the target host os-release
ID=debian
VERSION_ID=13
# or, when the distro defines it:
# SYSEXT_LEVEL=1.0
ARCHITECTURE=x86-64

Architecture identifiers follow ConditionArchitecture= style names (x86-64, arm64, …), not always raw uname -m.

Lab: confext for a reversible /etc overlay

NAME="ssh-banner-lab"
ROOT="/var/lib/confexts/${NAME}"

sudo rm -rf "$ROOT"
sudo mkdir -p \
  "$ROOT/etc/extension-release.d" \
  "$ROOT/etc"

sudo tee "$ROOT/etc/extension-release.d/extension-release.${NAME}" >/dev/null <<'EOF'
ID=_any
ARCHITECTURE=_any
EOF

# Additive file preferred. Shadowing existing files works via overlayfs
# but is discouraged for general packaging.
sudo tee "$ROOT/etc/ssh/banner.sysext" >/dev/null <<'EOF'
Authorized access only — confext lab banner
EOF

systemd-confext list
sudo systemd-confext merge   # or refresh
ls -l /etc/ssh/banner.sysext
findmnt /etc

sudo systemd-confext unmerge

Confext merges mount /etc with nosuid and, by default, noexec (override with --noexec=false only if you understand the trade-off).

Boot integration

Enable the oneshots/services your distro ships:

systemctl status systemd-sysext.service systemd-confext.service
sudo systemctl enable --now systemd-sysext.service
# enable confext only if you use confexts
sudo systemctl enable --now systemd-confext.service

These are guaranteed to finish before basic.target, so normal services can rely on merged files under /usr, /opt, and /etc.

Early-boot / initrd helpers also exist:

• systemd-sysext-initrd.service / systemd-confext-initrd.service
• systemd-sysext-sysroot.service / systemd-confext-sysroot.service

Sysroot helpers matter when you need extension content visible to very early consumers (for example some systemd-sysusers definitions). Note the documented limitation: with a split /var, extensions under /sysroot/var/lib/extensions may not merge in the sysroot pass and are handled later by the main OS services.

Mutability modes (systemd 256+)

Default merge on a writable host freezes the overlaid base directory as read-only for as long as extensions remain merged. That surprises people on classic package-managed systems.

Modes from systemd-sysext(8) / UAPI.4:

Write routing (non-ephemeral):

• /usr writes → /var/lib/extensions.mutable/usr/
• /opt writes → /var/lib/extensions.mutable/opt/
• /etc writes → /var/lib/extensions.mutable/etc/

To keep the real host tree writable while merged, symlink routing targets back:

sudo mkdir -p /var/lib/extensions.mutable
sudo ln -sfn /usr /var/lib/extensions.mutable/usr
sudo ln -sfn /opt /var/lib/extensions.mutable/opt
sudo ln -sfn /etc /var/lib/extensions.mutable/etc

# one-shot:
sudo systemd-sysext refresh --mutable=auto
# or persist via sysext.conf / confext.conf (Mutable=)

Persistent config (see sysext.conf(5)):

# /etc/systemd/sysext.conf.d/20-mutable.conf
[SysExt]
Mutable=auto

# /etc/systemd/confext.conf.d/20-mutable.conf
[ConfExt]
Mutable=auto

Mutable= in conf files is a relatively new knob (documented around systemd 259 in current man pages); CLI --mutable= landed in 256. Check systemd-sysext --version on your hosts.

Disk images, verity, and image policy

Directory trees are perfect for labs. Production immutable fleets usually want *.raw images (often erofs/squashfs) with optional verity, built in the same pipeline as the base OS (mkosi is a common choice).

When operating on disk images, systemd enforces an image policy (systemd.image-policy(7)). Defaults roughly allow root/usr with verity/signed/encrypted/unprotected/absent combinations for sysext; initrd /.extra/sysext/ defaults are stricter (signed oriented). Override with:

sudo systemd-sysext merge --image-policy='root=verity+signed:usr=verity+signed'

Tighten this deliberately on production appliances.

Operational patterns that work well

1. Optional debug/toolchain layer on immutable hosts

Ship gdb, strace, perf, and friends as a signed sysext. Merge when diagnosing; unmerge when done. Base image stays minimal.

2. Local rebuild of one component

From the man page’s own example pattern:

sudo make DESTDIR=/var/lib/extensions/mytest install
sudo systemd-sysext refresh
# exercise /usr paths as if installed into the OS
sudo rm -rf /var/lib/extensions/mytest
sudo systemd-sysext refresh

3. confext feature flags

Bake /etc drop-ins for a service into a confext. Deploy the image, systemd-confext refresh, restart the unit. Remove the confext to make the old config disappear with the overlay — no leftover drop-in archaeology.

4. UKI companion sysext in the ESP

systemd-stub can expose extension images from the ESP into /.extra/sysext/ for initrd/early use. Pair this with UKI workflows when the extension must be available before the real root’s /var is online.

Verification checklist

systemd-sysext list
systemd-sysext status
systemd-confext list
systemd-confext status

findmnt /usr /opt /etc
mount | grep -E 'overlay|sysext|confext' || true

# After merge, confirm additive path and that host identity is intact
test -e /usr/lib/os-release && grep ^ID= /usr/lib/os-release

Rollback and recovery

What not to use this for

• Replacing apt/dnf/pacman — no dependencies, no file conflict database, no updates channel of its own
• Untrusted third-party code — zero isolation; treat sysext like installing into the OS
• Shipping sandboxed long-running services — use portablectl instead
• A/B OS updates — that is systemd-sysupdate / image slots territory
• Per-file integrity of arbitrary mutable paths — look at fs-verity / dm-verity designs instead

Minimal production recipe

# 1) Build extension tree or .raw in CI alongside the base image
# 2) Install to the host
sudo install -d /var/lib/extensions
sudo cp dist/devtools.raw /var/lib/extensions/devtools.raw
# directory form also fine:
# sudo rsync -a dist/devtools/ /var/lib/extensions/devtools/

# 3) Merge now and on boot
sudo systemctl enable systemd-sysext.service
sudo systemd-sysext refresh

# 4) Confirm
systemd-sysext status
findmnt /usr

For confext, mirror the same flow under /var/lib/confexts/ and systemd-confext.

References

• systemd-sysext(8) — merge/unmerge/refresh, mutability, search paths, initrd/sysroot services
• sysext.conf(5) — Mutable=, ImagePolicy= drop-ins
• os-release(5) / extension-release — identity and matching rules
• UAPI.4 Extension Images — sysext vs confext, ordering, mutability model
• systemd.image-policy(7) — disk image admission policy
• Portable Services documentation (linked from the sysext man page) — isolation contrast

Closing

systemd-sysext and systemd-confext give you a first-class, reversible way to layer files onto /usr, /opt, and /etc without pretending to be a package manager or a container. Keep extensions additive, match extension-release carefully, enable mutability only when you mean to, and pick portable services when you need isolation instead of a host-tree merge.

Once you have a base image you trust, most “just one more binary” problems stop being full rebuilds — they become an extension refresh.