Apps with PVC Stuck in Deploying and Can't Start
Description
Problem/Justification
Impact
duplicates
Activity
Waqar Ahmed December 1, 2021 at 6:35 PM
in that case we can close this ticket as that was the only outstanding issue i was concerned with. Thanks for bringing that up again
Rick Bollar December 1, 2021 at 5:48 PMEdited
I can tell you that since applying a fix suggested by the system now reboots correctly and apps start upon reboot.
zfs set mountpoint=legacy $(zfs list -o name | grep ix-applications | grep pvc)
Kjeld Schouten-lebbing December 1, 2021 at 5:42 PM
Interesting, I assumed that was what above PR (also) fixed.
Thanks for noting
Waqar Ahmed December 1, 2021 at 5:35 PM
those mountpoints shouldn't be reset on reboot which is happening for Rick as he mentions in his description.
Kjeld Schouten-lebbing December 1, 2021 at 5:24 PMEdited
Why do you need to do that ?
You just merged the fix...
This is literally the fix for this issue, afaik:
https://github.com/truenas/middleware/pull/7914
So this is a duplicate of https://ixsystems.atlassian.net/browse/NAS-113447#icft=NAS-113447
All apps with a PVC are stuck in "Deploying" and never start. The Application Events Log shows the following, which suggests there's a mountpoint problem:
2021-11-18 23:42:53
MountVolume.SetUp failed for volume "pvc-2e971231-3c61-4d34-969b-70f60029c7fb" : rpc error: code = Internal desc = rpc error: code = Internal desc = zfs: mount failed err : not able to mount, cannot mount '/mnt/var/lib/kubelet/pods/5927b158-0ba2-46d8-9e8a-4c70c8471aad/volumes/kubernetes.io~csi/pvc-2e971231-3c61-4d34-969b-70f60029c7fb/mount': failed to create mountpoint: Read-only file system
Changing the mountpoint manually to "legacy" was suggested by @Ornias1993 and it resolves the issue temporarily e.g. if you apply a command similar to this:
root@scale[~]# zfs set mountpoint=legacy ixapps/ix-applications/releases/truecommand/volumes/pvc-7d9e5ab6-e00d-4240-a795-08edf93baac0
...the app will start again.
Unfortunately, a reboot reverts the change back to the broken state.
This affects new app installations, as well as existing app installations, so simply reinstalling the apps does not resolve the issue and the system seems to be in a permanently broken state.
The system got into this state while reinstalling TrueNAS and restoring a saved configuration file.
In the attached debug, the TrueCommand & Plex apps have had the above workaround applied. Mealy was installed after this issue began. All other apps are in their pre-reinstallation state.
IF there's a way to resolve the issue without a complete new system reinstallation, I'd appreciate the advice.