Apps with PVC Stuck in Deploying and Can't Start

Description

All apps with a PVC are stuck in "Deploying" and never start. The Application Events Log shows the following, which suggests there's a mountpoint problem:

2021-11-18 23:42:53
MountVolume.SetUp failed for volume "pvc-2e971231-3c61-4d34-969b-70f60029c7fb" : rpc error: code = Internal desc = rpc error: code = Internal desc = zfs: mount failed err : not able to mount, cannot mount '/mnt/var/lib/kubelet/pods/5927b158-0ba2-46d8-9e8a-4c70c8471aad/volumes/kubernetes.io~csi/pvc-2e971231-3c61-4d34-969b-70f60029c7fb/mount': failed to create mountpoint: Read-only file system

Changing the mountpoint manually to "legacy" was suggested by @Ornias1993 and it resolves the issue temporarily e.g. if you apply a command similar to this:

root@scale[~]# zfs set mountpoint=legacy ixapps/ix-applications/releases/truecommand/volumes/pvc-7d9e5ab6-e00d-4240-a795-08edf93baac0

...the app will start again.

Unfortunately, a reboot reverts the change back to the broken state.

This affects new app installations, as well as existing app installations, so simply reinstalling the apps does not resolve the issue and the system seems to be in a permanently broken state.

The system got into this state while reinstalling TrueNAS and restoring a saved configuration file.

In the attached debug, the TrueCommand & Plex apps have had the above workaround applied. Mealy was installed after this issue began. All other apps are in their pre-reinstallation state.

IF there's a way to resolve the issue without a complete new system reinstallation, I'd appreciate the advice.

Problem/Justification

None

Impact

None

Activity

Show:

Waqar Ahmed December 1, 2021 at 6:35 PM

in that case we can close this ticket as that was the only outstanding issue i was concerned with. Thanks for bringing that up again  

Rick Bollar December 1, 2021 at 5:48 PM
Edited

  I can tell you that since applying a fix suggested by  the system now reboots correctly and apps start upon reboot.

zfs set mountpoint=legacy $(zfs list -o name | grep ix-applications | grep pvc)

Kjeld Schouten-lebbing December 1, 2021 at 5:42 PM

Interesting, I assumed that was what above PR (also) fixed.
Thanks for noting

Waqar Ahmed December 1, 2021 at 5:35 PM

those mountpoints shouldn't be reset on reboot which is happening for Rick as he mentions in his description.

Kjeld Schouten-lebbing December 1, 2021 at 5:24 PM
Edited

Why do you need to do that ?
You just merged the fix...

This is literally the fix for this issue, afaik:
https://github.com/truenas/middleware/pull/7914

So this is a duplicate of https://ixsystems.atlassian.net/browse/NAS-113447#icft=NAS-113447

Duplicate
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Time remaining

0m

Components

Fix versions

Affects versions

Priority

Katalon Platform

Created November 19, 2021 at 1:19 PM
Updated April 19, 2023 at 1:22 PM
Resolved December 1, 2021 at 6:35 PM