128byte memory leak in ZFS

Lets guess for now it is already fixed speculative prefetcher issue.

 did you get a chance to try with 12.0-U3?

In the new debug in `vmstat -m` I see >10GB of RAM consumed by ex-Solaris code, likely ZFS in 128 byte chunks:

Unfortunately I can't guess what it is just from that line.  It could help if you find out what exactly workload makes it to grow that fast.  And really make sure to upgrade in case it is something that was already fixed.  We expect TrueNAS 12.0-U3 to be released tomorrow.

Hi, I haven't had the chance to upgrade, but I think this is heading back into a tailspin.

I think this latest began at 4 April, 00:00.  Out of swap space again, wasn't doing anything special other than file serving.  New debugs attached.  Have also attached some syslogs, and a graph of swap usage from Zabbix.

I also now have no web interface access, and several process in D state:

I'm sorry for the delay.  Unfortunately in the provided debug I see no logs before the last reboot.  Not sure what happened to them, considering that now the system dataset seems to be stored on the data pool.  But any way I can't say anything about hang, neither I see any swap-related errors.  Please update to the latest TrueNAS 12.0-U2.1 and then report the problem again with the debug that hopefully show anything.