Mass segfaults on a ZFS host: not bad RAM, an unsigned underflow in zfs_fillpage()

Short answer: unrelated processes segfaulting all at once on a ZFS host, plus systemd[1]: Freezing execution in the journal, is kernel arithmetic rather than failing memory. In OpenZFS before 2.4.4 and 2.3.9, zfs_fillpage() computed the read length as io_len = i_size - io_off on unsigned types. Truncate a file in place at the moment an mmap page is faulted in, i_size drops below io_off, the difference underflows to near 2⁶⁴, and dmu_read() zero-fills physical memory far past the page it was asked for. The machine survives that. What finishes it off is systemd, whose CrashAction defaults to freeze.

From outside it looked like a powered-off box: no SSH, no HTTP, no hypervisor UI. A hard reset from the provider's panel brought it back, which left the question nobody enjoys: what that was, and whether it happens again tomorrow.

An hour of digging produced two answers instead of one. The ZFS bug is the trigger. The price of the failure was set by a systemd default.

TL;DR

• Cause: unsigned underflow in zfs_fillpage() when an mmap read races an in-place truncate. The kernel zero-fills physical pages belonging to whatever else was there, so the victims are random processes.
• Fixed in: OpenZFS 2.4.4 and 2.3.9, both published 21 August 2026. On Proxmox, zfsutils-linux 2.4.4-pve1.
• Not fixed in: 2.2.11, published the same minute as the other two. Verified by reading the code in the tag, not the release notes.
• Why the outage runs for hours: CrashAction=freeze is the systemd default. A crashed PID 1 hangs forever, nothing restarts services, systemctl blocks, sshd dies later with nobody to bring it back.
• One-minute test against bad RAM: ECC counters in /sys/devices/system/edac/mc/mc0/ at zero and no Machine Check in the journal means the DIMMs are not your problem.
• Tell in the dump: the length register of memset is not garbage, it is a tidy negative number, exactly minus another register. That is a size computation gone wrong, not failing hardware.
• One-line mitigation: CrashAction=reboot in a drop-in turns a repeat from an overnight outage into a 10-second one.

Why the machine looks powered off when it is not

It looks identical to a powered-off machine from outside, which is why the first hypothesis is always wrong. No port answers, the provider's panel reports power as normal, and support has nothing to add. The obvious theory is that the box went down, so you start looking for the cause of a shutdown: an ACPI event, a watchdog, a power dip in the rack.

The journal kills that theory. journalctl --list-boots shows the previous boot, and its entries run right up to a point a couple of minutes before the new boot began, which is the board's POST. The last line before the reset was an nginx worker dying:

nginx[…]: [alert] …: worker process … exited on signal 11

The machine never powered off. It ran the whole time and kept writing to the journal that it was in trouble. Nobody outside could hear it.

That yields a second, nastier fact. The moment things actually broke sits much earlier than the "crash time" you get from last. last -x will show crash without saying when it started. The real start is the first segfault in the avalanche, and it can be a whole night before the reset.

Why it is not bad RAM, even though it looks exactly like bad RAM

Simultaneous segfaults across unrelated processes is the classic signature of a failing DIMM, and the reflex it triggers is the wrong one. Before you schedule downtime for memtest86+, spend a minute on two counters:

cat /sys/devices/system/edac/mc/mc0/ce_count   # corrected errors
cat /sys/devices/system/edac/mc/mc0/ue_count   # uncorrected
journalctl -b -1 -k | grep -i "machine check"

Zeros in all three places on an ECC system mean memory as a subsystem is healthy. A single flipped bit would have been caught and logged. While you are there, confirm ECC is actually active rather than merely printed on the invoice:

dmidecode -t memory | grep -E "Error Correction Type|Total Width|Data Width"

Total Width needs to be 8 bits wider than Data Width, 72 against 64, because those are the physical bits doing the correction. Equal widths mean you have no ECC, whatever the spec sheet claims.

That minute saves hours of downtime spent diagnosing the wrong subsystem. ECC protects against flipped bits, not against code that walks the ordinary execution path and writes zeros into the wrong place.

Four dead ends, all through the same function

Every hypothesis that came from searching the symptom was wrong, and they cost more time than the answer did. Writing them out because each one closes a direction: if your stack looks like this, you can skip these branches.

Dead end 1: a Direct IO race. The first sensible search on the call stack landed on a real race in OpenZFS, where a concurrent O_DIRECT write can set db->db_data = NULL and dmu_read_impl() then faults on the dereference. The function names matched mine exactly. Checking the tunable made it look even better:

# cat /sys/module/zfs/parameters/zfs_dio_enabled
1

I said out loud that this was the leading theory. The registers disagreed: what I had was not a NULL dereference but a memset with a sensible destination and an absurd length. Different mechanism. I should have checked that in the first minute, since the registers were on screen the whole time.

Dead end 2: SLUB corruption on encrypted ZFS. Issue #18427 describes my symptom nearly word for word — mass userspace segfaults, reproducible across several releases. One command ended it:

# zfs get -r encryption rpool
NAME    PROPERTY    VALUE    SOURCE
rpool   encryption  off      default

Dead end 3: an already-fixed regression in dmu_read_impl. Issue #17886 is an oops in the same function, also with memory clobbered by a bad length, closed by PR #17915. It read like "known bug, fix exists, go upgrade." Then I read the PR: it restores lost parentheses in the BRT_RANGESIZE_TO_NBLOCKS() macro, and the author states the blast radius plainly — "this could cause small memory corruptions for vdevs bigger than 64TB+". A web-workload pool of a few terabytes is an order of magnitude short of that threshold. The stack in that bug also runs through brt_load() during pool import rather than through mmap under load. Same crash site, different bug.

Dead end 4: "the server was off for hours." Covered above. It died against a continuous journal from the previous boot.

The unpleasant summary: all four trails ran through dmu_read_impl. It is a popular function, "memory corruption" is far too general a symptom, and searching by symptom in that situation returns an endless feed of other people's misfortunes. The break came from arithmetic, not from searching.

The giveaway: the memset length is exactly minus another register

The whole answer sits in the oops dump, and it only needs counting. Trimmed to what matters:

BUG: unable to handle page fault for address: ffff8983e8400000
#PF: supervisor write access in kernel mode
#PF: error_code(0x0003) - permissions violation
Comm: php-fpm
RIP: 0010:memset+0xb/0x20
RAX: 0000000000020000   RSI: 0000000000000000   RDI: ffff8983e8400000
RDX: ffffffffff7a4000   R15: 000000000085c000
Call Trace:
 ? dmu_read_impl+0x21e/0x230 [zfs]
 dmu_read [zfs]
 zfs_fillpage [zfs]
 zfs_getpage [zfs]
 zpl_read_folio [zfs]
 filemap_read_folio
 filemap_fault
 __do_fault
 handle_mm_fault

Read the memset arguments. Under the x86-64 SysV ABI those are RDI for destination, RSI for the fill byte, RDX for the count.

RSI = 0, so it is zero-filling. RDX = 0xffffffffff7a4000 looks like garbage at a glance. Now take R15 = 0x85c000 = 8,765,440. This is the part worth slowing down for: 0xffffffffff7a4000 is exactly −8,765,440 in two's complement. Not a random value, not a clobbered register. A tidy negative number.

A negative length does not appear from nowhere. It means something subtracted a larger number from a smaller one in unsigned arithmetic. The culprit is not whoever corrupted memory, it is whoever computed the size.

RAX = 0x20000 fits the picture too: 128 KiB, the default ZFS recordsize. An ordinary record read whose length went backwards.

With that hypothesis the search stopped being a search by symptom and became a search by function name plus exact version. The right issue was the first result.

What is broken in ZFS: an invariant that does not survive the release build

Two lines in zfs_fillpage() do the damage, and the guard in front of them does not exist in release builds. Here is the code from the 2.4.3 tag:

loff_t i_size = i_size_read(ip);
u_offset_t io_off = page_offset(pp);
size_t io_len = PAGE_SIZE;

ASSERT3U(io_off, <, i_size);

if (io_off + io_len > i_size)
        io_len = i_size - io_off;

The invariant is stated in plain sight: the page offset must be smaller than the file size. One line down, a subtraction is built on it. All honest, right up until you open the definition of ASSERT3U for a non-debug build in include/os/linux/spl/sys/debug.h:

/*
 * Debugging disabled (--disable-debug)
 */
#ifdef NDEBUG

#define ASSERT3U(x, y, z)                                               \
        ((void) sizeof ((uintptr_t)(x)), (void) sizeof ((uintptr_t)(z)))

Two sizeof expressions that check nothing. The header says so directly: assertions are "compiled out when NDEBUG is defined, this is the default behavior for the SPL." Distributions, Proxmox included, build ZFS that way. So the invariant is documented, and unenforced exactly where enforcement would have mattered.

Truncating the file in place while a page is being faulted in breaks that invariant. i_size drops below io_off, the subtraction underflows, and io_len becomes a number close to 2⁶⁴. dmu_read() takes it at face value and fills.

The fix adds the runtime check the assertion pretended to be. From the 2.4.4 tag:

    /*
     * The page may be faulted in after the file has been truncated.
     * There is no data to read; just zero-fill the page.
     */
    if (io_off >= i_size) {
        void *zva = kmap(pp);
        memset(zva, 0, PAGE_SIZE);
        kunmap(pp);
        ClearPageError(pp);
        SetPageUptodate(pp);
        return (0);
    }

The commit is 223b8bc446, "linux: handle mmap read beyond file size", merged 30 June 2026 by Brian Behlendorf. Its trailers are worth reading: Reported-by: Iliya Polihronov (@vnsavage) (Automattic), Reviewed-by: Alexander Motin. WordPress.com hit this independently, which fits — it is a PHP-shaped bug. readfile() on a cache file that another request is rewriting in place is the exact race.

The crashing process in my dump is php-fpm, running under an unprivileged LXC. Its uid maps to 82 inside the container, which is www-data in Alpine, so it was PHP from an Alpine image in Docker inside LXC, on a Debian host. Four layers down from the kernel, and none of them at fault.

Which versions are actually fixed, and which release from the same day is not

Three OpenZFS releases went out on 21 August 2026, and the fix is in two of them. Checking release notes is not enough here, so I read zfs_vnops_os.c in each tag:

The three tags were published within about a minute of each other. Release notes for 2.4.4 and 2.3.9 both carry the line linux: handle mmap read beyond file size #18715; the notes for 2.2.11 do not. In the 2.2.11 tree, ASSERT3U(io_off, <, i_size); and the bare subtraction are still there. The same holds in the 2.1 branch, so this is an old bug rather than a fresh regression, and downgrading to an older kernel does not help.

Check your own build rather than trusting a version number:

zfs version
# then, for the tag you are on:
curl -sf "https://raw.githubusercontent.com/openzfs/zfs/zfs-$(zfs version | head -1 | cut -d- -f2 | cut -d_ -f1)/module/os/linux/zfs/zfs_vnops_os.c" \
  | grep -c 'io_off >= i_size'

Nonzero means the guard is in. Use curl -sf: without -f, a nonexistent tag returns a 404 page that greps to "0 matches", which is indistinguishable from "the fix is missing." I nearly recorded that as a result once.

On Proxmox the fixed package is already in the ordinary repository. In pve-no-subscription for trixie, zfsutils-linux is at 2.4.4-pve1, so this is a plain apt upgrade plus a reboot rather than anything exotic.

Upstream the story is not closed. Issue #18787, opened 12 July 2026 by the Incus maintainer, who was losing hosts full of LAMP containers, is still open today even though the fix landed two weeks before it was filed. Track the PR number, not the ticket state. A regression test followed on 18 July.

Why the outage lasts hours: CrashAction defaults to freeze

The bug corrupts memory in milliseconds; the outage lasts all night because of a systemd default. Milliseconds after the corruption started, PID 1 caught its own crash:

systemd[1]: Caught <ABRT>, from our own process.
systemd[1]: Caught <ABRT>, dumped core as pid …
systemd[1]: Freezing execution.

CrashAction= arrived in systemd 256, alongside the systemd.crash_action= kernel command line option. The manual is unambiguous about the default:

Takes one of "freeze", "reboot" or "poweroff". Defaults to "freeze". If set to "freeze", the system will hang indefinitely when the system manager (PID 1) crashes. If set to "reboot", the system manager (PID 1) will reboot the machine automatically when it crashes, after a 10s delay.

Freezing makes sense on a desktop, or anywhere an operator is standing next to the console and wants the corpse preserved. On rented hardware in someone else's data centre it means "hang forever and wait for a human to notice."

With PID 1 frozen the system loses the ability to repair itself. Nothing restarts crashed services, systemctl blocks, a clean shutdown is impossible. sshd went next, and after its last line there is not one further mention of it in the journal: no login attempts, no refusals. Port 22 simply had nobody listening. nginx meanwhile kept crashing at the same instruction pointer, tens of thousands of times, roughly twice a second, all night. The master was alive and forking workers; each worker inherited poisoned memory and died at once. A restart would have fixed it, and there was no one left to issue one.

The mitigation is one drop-in file:

# /etc/systemd/system.conf.d/10-crash-reboot.conf
[Manager]
CrashAction=reboot

systemd-analyze cat-config systemd/system.conf   # inspect before applying
systemctl daemon-reexec                          # apply to the running PID 1

A repeat now costs ten seconds instead of a night. That is the whole trade: I cannot remove the bug from a release that does not contain the fix, so I bought recovery time instead of prevention.

Side gotchas worth stealing

Two smaller findings came out of the same morning, and both are worth checking on any Docker host.

A restart: on-failure policy survives a crash but may not survive a clean reboot. The containers came back after the hard reset because their processes were killed by SIGKILL, so the exit code was nonzero and the policy fired. On a graceful reboot dockerd sends SIGTERM, and anything that exits 0 (nginx does) is not a failure, so on-failure leaves it down. The configuration survives the disaster and can fail the maintenance window. Switching everything to unless-stopped with docker update fixes it live, no recreation and no downtime.

A container with no restart: line at all defaults to no. One monitoring container was in that state, so after every reboot it stayed down, and the proxy that depended on it spun through dozens of restarts inside half an hour with host not found in upstream. depends_on does not help here: it orders docker compose up and has no effect when the daemon starts containers on boot.

What to do right now

1. Check whether your ZFS build carries the guard: zfs version, then look for io_off >= i_size in zfs_fillpage() for that tag. Anything on 2.2.x or 2.1.x is exposed, including 2.2.11 from 21 August 2026.
2. Upgrade to 2.4.4 or 2.3.9. On Proxmox trixie that is zfsutils-linux 2.4.4-pve1 from pve-no-subscription, then a reboot into the new module.
3. If you cannot upgrade yet, cut the recovery time: drop in CrashAction=reboot and run systemctl daemon-reexec. Ten seconds against an open-ended hang.
4. Add an availability check from outside the host. Mine was found by a person arriving at work, which is a bigger hole than the kernel bug.
5. Audit Docker restart policies: docker inspect -f '{{.Name}} {{.HostConfig.RestartPolicy.Name}}' $(docker ps -aq). Anything showing no or on-failure is a machine that may not come back from a planned reboot.

Bottom line

Three layers of blame, and you can only fix two of them. The ZFS underflow is upstream's to fix, and as of today it is fixed in 2.4.4 and 2.3.9 while 2.2.11 from the same day still carries it. The systemd default turned a memory-corruption event into an overnight outage, and that costs one line of configuration to change. The missing availability check is what stretched the outage until morning, and that one is nobody's fault but mine.

The transferable part is the diagnostic move. When the kernel faults inside memset or memcpy, look at the length register before you look at anything else, and check whether it is a tidy negative number. If it is, stop investigating the hardware: someone computed a size wrong, and the fastest path to the answer is the function names in the stack plus your exact version, not the symptom.

Primary sources: commit 223b8bc446 and PR #18715 in OpenZFS, issue #18787, the zfs-2.4.4 release, and systemd(1) for systemd.crash_action=.

Originally published at bitpage.me — BitPage, a technical blog on backend, databases, infrastructure and incident post-mortems.