Periodic Snapshot Tasks do not execute after upgrade to SCALE 22.12.0 Bluefin

This issue has now been closed. Comments made after this point may not be viewed by the TrueNAS Teams. Please open a new issue if you have found a problem or need to re-engage with the TrueNAS Engineering Teams.

running that command to remove systemd.unified_cgroup_hierarchy=0 from kernel_extra_options and rebooting allowed my periodic snapshot tasks to run successfully at the scheduled times. I do not know how that kernel option got added originally (it’s possible I did it manually but I don’t recall doing so, and I don’t have the knowledge to muck around with kernels generally), but removing it has fixed this periodic snapshot task issue. Thanks!

What is more, my apps started without problem after this reboot as well, so that’s good!

running midclt call system.advanced.update '{"kernel_extra_options": ""}' will unset those kernel parameters and after a reboot you should be able to test if that works or not..if it still does not, please reboot one more time and grab a debug right after reproducing the problem (motivation for doing another reboot is solely to make sure the changes have come into effect at grub level in case 1 single reboot is not enough)

Ooh, this gets interesting! To answer your question, no I had not previously manually added that to the kernel boot parameters. This system was upgraded over several versions of Angelfish, and then to Bluefin.

However, in the time since I opened this ticket and attached the debug from 2022-12-21, I upgraded my server hardware (CPU, mobo, and ram), reinstalled Bluefin 22.12 from scratch, and restored my config. At that point, my installed apps would not start. I was getting the exact k3s error reported in . From that ticket, I ran the suggested command:

This allowed my k3s apps to start. But my scheduled snapshot tasks still do not execute. I do not know if they would have executed after the reinstall but before I ran this command. I was focused on getting my apps up and running.

I will attach another debug from my current reinstalled system.

if you can provide the command, I will revert back to cgroups v2 and see if that allows the snapshot tasks to execute (hopefully it doesn’t break my apps again).

can you confirm if you manually added systemd.unified_cgroup_hierarchy=0 to kernel boot parameters?

I am seeing

In bluefin we only have cgroups v2 available so this should not be there..