Core files found for Python 3.9

@FameWolf thanks for the core dumps. I've analyzed them and I've fixed this particular problem. The original ticket is https://ixsystems.atlassian.net/browse/NAS-113251#icft=NAS-113251 that has the URL to the commit that I pushed. Anyways, the good news is that the next SCALE release will have this fix in there. So when it's released to the public, I'd recommend upgrading to prevent these from occurring.

I reverted to the previous nights snapshot of ix-applications which seems to have gotten me back to where I started.

Checking docker on command line shows:

root@truenas/mnt/tank/ix-applications# service docker status
●docker.service - Docker Application Container Engine
    Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
    Active: active (running)since Fri 2021-11-12 22:46:03 EST; 38min ago
TriggeredBy: ●docker.socket
      Docs: https://docs.docker.com
  Main PID: 12904 (dockerd)
     Tasks: 90
    Memory: 136.6M
    CGroup: /system.slice/docker.service
            └─12904 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Nov 12 23:22:31 truenas.local dockerd[12904]: time="2021-11-12T23:22:31.726443571-05:00" level=error msg="Error removing mounted layer e03de7241123b08b479e6bbbb80693ba123e49c5d201f9c>
Nov 12 23:22:31 truenas.local dockerd[12904]: time="2021-11-12T23:22:31.726553867-05:00" level=error msg="Handler for DELETE /v1.41/containers/e03de7241123b08b479e6bbbb80693ba123e49c>
Nov 12 23:23:01 truenas.local dockerd[12904]: time="2021-11-12T23:23:01.794901608-05:00" level=error msg="Error removing mounted layer 05755d2c86c0cb4b4c87930cdd5093ea5cf587c01c8275e>
Nov 12 23:23:01 truenas.local dockerd[12904]: time="2021-11-12T23:23:01.795014174-05:00" level=error msg="Handler for DELETE /v1.41/containers/05755d2c86c0cb4b4c87930cdd5093ea5cf587c>
Nov 12 23:24:01 truenas.local dockerd[12904]: time="2021-11-12T23:24:01.870091601-05:00" level=error msg="Error removing mounted layer 7a639189ca6ba949e969163c38e68a588882098e2548d83>
Nov 12 23:24:01 truenas.local dockerd[12904]: time="2021-11-12T23:24:01.870275223-05:00" level=error msg="Handler for DELETE /v1.41/containers/7a639189ca6ba949e969163c38e68a588882098>
Nov 12 23:24:31 truenas.local dockerd[12904]: time="2021-11-12T23:24:31.940380547-05:00" level=error msg="Error removing mounted layer 05755d2c86c0cb4b4c87930cdd5093ea5cf587c01c8275e>
Nov 12 23:24:31 truenas.local dockerd[12904]: time="2021-11-12T23:24:31.940510044-05:00" level=error msg="Handler for DELETE /v1.41/containers/05755d2c86c0cb4b4c87930cdd5093ea5cf587c>
Nov 12 23:24:31 truenas.local dockerd[12904]: time="2021-11-12T23:24:31.970393060-05:00" level=error msg="Error removing mounted layer e03de7241123b08b479e6bbbb80693ba123e49c5d201f9c>
Nov 12 23:24:31 truenas.local dockerd[12904]: time="2021-11-12T23:24:31.970522720-05:00" level=error msg="Handler for DELETE /v1.41/containers/e03de7241123b08b479e6bbbb80693ba123e49c>

The "migration" finally timed out I guess as it's vanished and now NO apps show up in my installed apps list.

Going back to the "tank" pool also produced an error:

CRITICAL

Failed to configure kubernetes cluster for Applications: [EFAULT] Docker service is not running

2021-11-12 22:36:20 (America/New_York)

Rebooting showed no apps on tank pool as well now.

I created a new pool on a new ssd named "apps" and attempted to change the pool to it with the checkbox to migrate the apps.  I'm unsure if it's related to this python error or not but it's been hung for 30 minutes and the gigabyte done has not changed where previously it was.