Cache-unfriendly filesystem usage, memory fragmentation and ARC

[runderwo]

System information

Type	Version/Name
Distribution Name	Ubuntu/Debian
Distribution Version	LTS/latest
Kernel Version	6.8-6.11
Architecture	x86_64
OpenZFS Version	2.2.2-2.2.6

Describe the problem you're observing

After moderate uptime of a few weeks, when a program tries to read or index the whole filesystem or a large chunk of it, the system seizes up, becomes unresponsive to input/network for 15-20 minutes. Eventually it recovers to a sluggish but usable state (with the offending process still running, consuming core time and disk I/O) where a tool like atop can be used to observe lingering heavy free page scan activity, - despite up to 10GiB of free/avail memory! (Linux page cache has been zeroed by this time.)

ARC is maxed out at 97% (almost 50% of system RAM according to the default settings).

Examining /proc/buddyinfo, there are no free pages >= 1MiB in the steady state and can be even worse right after the "seizure" with no free pages >= 128KiB.

I suspect the partial recovery is thanks to kcompactd activity. I am thinking that ZFS should drop cached file blocks from ARC not just when the kernel low watermark is reached, but also when higher order free pages become exhausted.

Describe how to reproduce the problem

Simulate normal memory fragmentation on a host, including multiple hibernate/resume cycles, then run duplicity, tracker3-miner, or similar programs which ingest the whole filesystem in a cache-unfriendly and ZFS-unfriendly way while monitoring the situation with atop.

Include any warning/errors/backtraces from the system logs

Dec 22 16:09:02 desktop kernel: zfs: module license 'CDDL' taints kernel.
Dec 22 16:09:02 desktop kernel: Disabling lock debugging due to kernel taint
Dec 22 16:09:02 desktop kernel: zfs: module license taints kernel.
Dec 22 16:09:02 desktop kernel: calling  openzfs_init+0x0/0xce0 [zfs] @ 428
Dec 22 16:09:02 desktop kernel: ZFS: Loaded module v2.2.2-0ubuntu9.1, ZFS pool version 5000, ZFS filesystem version 5
[..]
Jan 22 14:05:06 desktop systemd-journald[935]: Under memory pressure, flushing caches.
[..]
Jan 22 14:16:31 desktop kernel: INFO: task chrome:3547537 blocked for more than 122 seconds.
Jan 22 14:16:31 desktop kernel:       Tainted: P           OE      6.8.0-51-generic #52-Ubuntu
Jan 22 14:16:31 desktop kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.```
[etc, etc]

[amotin]

Have you tried to set zfs_arc_shrinker_limit=0 after updating to the latest 2.2.x release? It was made default in 2.3.0: #16909 .

[runderwo]

I have not, but will try that out and report back after it's had some time to bake on a host with extended uptime. Thanks for the pointer!

[XyFreak]

Hi,

I have been suffering from the exact same issue ever since upgrading to zfs 2.2.0. I'd like to add a few things:

• When indexing starts, the system seems normal at first but then memory consumption spikes rapidly. My server has about 64GB of ram and looking at htop while running the operation, memory usage will spike suddenly and rapidly. It takes about 3 seconds and the system is entirely OOM and unresponsive.
• Terminating the process that indexes the filesystem before the system becomes unresponsive helps but the used memory does not seem to get freed anymore. A reboot is required to reclaim said memory.
• There is a similar issue with filesystem watches - except it takes days to a week or two to trigger instead of seconds. Might be unrelated but I thought I'd mention it.

I just reproduced it and had a watch on both arc_summary (notice the nice "Available memory size") and /proc/slabinfo running in parallel.

Note that I copied the frozen terminal output into a text file since the system was no longer responsive. I hope it contains the relevant information you might be looking for. Otherwise I can try to extract some information on a partially bricked system where indexing has been killed before the system becomes unresponsive.

arc_summary.txt
slabinfo.txt
slabinfo after reboot

zfs_arc_shrinker_limit is already 0 (I double checked) on my setup as I am running zfs 2.3.0 right now, Kernel 6.12.8.
zfs_arc_max does not seem to really matter here. I have tried values ranging from 16GB to 60GB (set via modprobe on boot, not during runtime) and it happens no matter what.

Another tidbit: I had htop running and while the green bar (used memory) was not at 100% (not even a third), the yellow one (cache) basically filled up the empty space. This would explain runderwos observation of there being "free memory" if memory used for caches was not taken into account. Edit: It was mentioned, I evidently just can't read, sorry

The issue is reproducible moments after the system has rebooted for me so no causing "memory fragmentation" required at all. If I can provide or try anything else, lmk.