Eighteen months of bad RAM, and the mail server that hid it

For eighteen months the Proxmox node in my house died every five to eight days. Not slowed down, not degraded: Plex, every guest and the web UI went at once, and the only way back was to walk over and hold the power button.

I lived with it for far too long, because a hard lockup on a home server looks like a thing that just happens. It is not. It is a symptom, and this one had been writing its diagnosis into the kernel log the entire time.

The verdict first

Failing non-ECC system memory. A hardware fault. Not Proxmox, not Plex, not the kernel.

The evidence was kernel slab corruption pinned to one address on one CPU: five faults over thirty-eight hours, in four unrelated processes, all landing in the same place. Software bugs do not do that. Bad memory does.

The fix was a new RAM kit. Everything else in this note is what I found while proving that, and most of it is worse than the RAM.

Why the box sat dead instead of rebooting

The host was configured to do nothing on a kernel fault:

kernel.panic            = 0    (hang forever, never reboot)
kernel.softlockup_panic = 0    (ignore soft lockups)
kernel.panic_on_oops    = 0

On one night in January the kernel detected soft lockups for eight straight minutes and, as instructed, did nothing about any of them. That is a hell of a default for a machine nobody is watching. It now panics and reboots.

The watchdog was no help either. Proxmox ships with softdog, which is a kernel timer, and a kernel timer cannot rescue a hard-locked kernel. It is useless for exactly this failure. The board's chipset has its own hardware watchdog that fires regardless of what the CPU is doing, and it is the one in use now.

The crash had a trigger

The lockups were not random. The scheduled SnapRAID sync ran at three in the morning, and the smoking gun was the same second:

03:00:01  CRON: (root) CMD (/usr/bin/snapraid sync)
03:00:01  BUG: unable to handle page fault ... CPU 6 ... Comm: sh

The shell that faulted was cron launching SnapRAID. A sync allocates half a gigabyte and churns the page cache across terabytes, which makes it by far the heaviest thing the box does, so it was the most likely thing to touch the bad memory. Worse, a parity sync running on corrupting RAM can write corrupt parity, which is the one thing parity must never be. The job was switched off until the RAM was replaced.

Why nobody noticed for eighteen months

This is the actual lesson.

Every one of those SnapRAID failures sent an email. None of them arrived. The mail service had been masked, symlinked to /dev/null, for a year and a half, and 157 messages were sitting in the queue:

80  Cron  /usr/bin/snapraid sync
77  Cron  /usr/bin/snapraid scrub -p 10

The box had been telling someone it was in trouble the whole time. It just had no way to say it out loud. Mail now goes out through a relay.

Everything else the sweep found

Once I was looking properly, the list got long:

• Plex had never been backed up. Not once. The backup job contained a PATH line and nothing else. There is now a scheduled backup to the parity disk, chosen because it adds no load to the array, and it has been restored from, which is the only test of a backup that counts.
• The hypervisor itself was not backed up either. Guest backups cover guests and nothing else, so if the NVMe died, the storage config, the mount table and the only record of which disk holds parity went with it. A nightly tarball of the host's own config fixed that, and it is tiny.
• A dead Ceph cluster was eating memory. Zero storage daemons, zero pools, on a single node: an abandoned experiment from early 2025 that could never have worked, still running and still costing about half a gigabyte on a host that was already short of memory. Stopped and disabled.
• The Plex container was allowed to take almost the whole machine. It was capped at 12 GB on a host with about 13 GB usable once the integrated GPU took its share. It never used that much, but it was permitted to, and memory pressure drives exactly the reclaim path that deadlocked this machine. It is capped at 8 GB now.
• The host was mounting its own shares onto themselves. Four network mounts pointed back at the same directories Samba exports, and failed on every single boot. Removed.
• The media pool would have filled one disk with seventeen terabytes free. mergerfs was on its default create policy, which only considers disks where the parent folder already exists. One disk held almost the whole library, so new albums would have landed there until it was full, and then the writes would have failed while the other disks sat empty. It now writes to whichever disk has the most free space.

What is still open

A postmortem that ends with "and now it is fine" is lying, so:

• Parity is stale. The sync was switched off for good reason, and the array has not had a full resync since January. Until it does, parity protects nothing added since then. The /homelab page says so.
• Offsite backup is not built yet. It will be restic to object storage. The consumer backup plan I already had turned out to be unable to back up a Linux server at all, which is the kind of thing you only learn by trying.
• The auto-recovery has not been proven by a real crash. It is configured and verified as present. Present is not the same as working.
• The long-term fix is ECC. This host ran with corrupting memory for eighteen months and produced zero machine check exceptions, because non-ECC memory has no way to say it is wrong. The next platform will have memory that can.

The lessons

1. A hard lockup is a symptom. Read the kernel log before you accept it as weather.
2. Alerting is part of the system. A box that cannot tell you it is failing is failing silently, and silence looks exactly like health.
3. A backup nobody has restored from is a hypothesis.
4. Defaults are decisions somebody else made for a different machine. Check the panic settings, the watchdog and the storage policies on anything you intend to leave alone.

The whole thing was diagnosed in one long session with Claude Code on the host, with every finding filed as an issue and the evidence pasted in. The RAM was the cause. The masked mail service was the reason it took eighteen months.