k3s/k8s Plex Failure

thank you for reaching out. Can you please create a new ticket and attach a debug as well ? We can continue from there. Thank you

Due to the power in the country that I'm in this issue has happened again. I'm saving off a debug log but it takes 20 minutes.
2021-07-01 22:37:03
MountVolume.SetUp failed for volume "kube-api-access-m6wcl" : [failed to fetch token: serviceaccounts "default" is forbidden: User "system:node:ix-truenas" cannot create resource "serviceaccounts/token" in API group "" in the namespace "ix-truenas-scale-plex": no relationship found between node 'ix-truenas' and this object, failed to sync configmap cache: timed out waiting for the condition]
2021-07-01 22:37:02
MountVolume.SetUp failed for volume "plex-probe-check" : failed to sync configmap cache: timed out waiting for the condition
2021-07-01 22:37:16
Failed to update endpoint ix-truenas-scale-plex/truenas-scale-plex-tcp: Operation cannot be fulfilled on endpoints "truenas-scale-plex-tcp": the object has been modified; please apply your changes to the latest version and try again
2021-07-01 22:37:16
Failed to update endpoint ix-truenas-scale-plex/truenas-scale-plex-udp: Operation cannot be fulfilled on endpoints "truenas-scale-plex-udp": the object has been modified; please apply your changes to the latest version and try again
2021-07-01 22:37:01
network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
2021-07-01 22:42:21
Scaled down replica set truenas-scale-plex-85db8c5b99 to 0
2021-07-01 22:42:21
Deleted pod: truenas-scale-plex-85db8c5b99-2w4j2
2021-07-01 22:43:12
Scaled up replica set truenas-scale-plex-7567df94f7 to 1
2021-07-01 22:43:12
Created pod: truenas-scale-plex-7567df94f7-jkwdc
 
0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
2021-07-01 22:46:29
Scaled down replica set truenas-scale-plex-7567df94f7 to 0
2021-07-01 22:46:29
Deleted pod: truenas-scale-plex-7567df94f7-jkwdc
2021-07-01 22:46:30
Created pod: truenas-scale-plex-98f9ffdfc-pswxz
 
0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
 
0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
2021-07-01 22:56:59
Scaled down replica set truenas-scale-plex-98f9ffdfc to 0
 
skip schedule deleting pod: ix-truenas-scale-plex/truenas-scale-plex-98f9ffdfc-pswxz
2021-07-01 22:56:59
Deleted pod: truenas-scale-plex-98f9ffdfc-pswxz
2021-07-01 22:46:30
Scaled up replica set truenas-scale-plex-98f9ffdfc to 1
2021-07-01 22:57:16
Created pod: truenas-scale-plex-98f9ffdfc-tr54x
 
0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
 
0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
 
truenas# k3s kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
ix-chia1 chia1-ix-chart-997d88d9-tnkth 1/1 Running 3 7d9h
ix-truenas-scale-plex truenas-scale-plex-98f9ffdfc-tr54x 0/1 Pending 0 14m
kube-system coredns-7448499f4d-lth7l 0/1 Pending 0 10m
kube-system openebs-zfs-controller-0 0/5 Pending 0 10m

truenas# k3s kubectl describe node ix-truenas
Name: ix-truenas
Roles: control-plane,master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=ix-truenas
kubernetes.io/os=linux
node-role.kubernetes.io/control-plane=true
node-role.kubernetes.io/master=true
openebs.io/nodeid=ix-truenas
openebs.io/nodename=ix-truenas
Annotations: csi.volume.kubernetes.io/nodeid: {"zfs.csi.openebs.io":"ix-truenas"}
k3s.io/node-args:
["server","--flannel-backend","none","--disable","traefik,metrics-server,local-storage","--disable-kube-proxy","--disable-network-policy",...
k3s.io/node-config-hash: W5IKNGASPB6HPORG7PXAIND2UBRVDHO7663Q4FLHOCAP4FXXOQUQ====
k3s.io/node-env: {"K3S_DATA_DIR":"/mnt/ssd-pool/ix-applications/k3s/data/1fda8eac79455ae721508123989e095a50c209cf7965df5630549292f7916941"}
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Sat, 24 Apr 2021 09:04:18 +0700
Taints: node.kubernetes.io/not-ready:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: ix-truenas
AcquireTime: <unset>
RenewTime: Thu, 01 Jul 2021 23:12:06 +0700
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message

------ ----------------- ------------------ ------ -------
MemoryPressure False Thu, 01 Jul 2021 23:10:04 +0700 Sat, 24 Apr 2021 09:04:18 +0700 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 01 Jul 2021 23:10:04 +0700 Sat, 24 Apr 2021 09:04:18 +0700 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 01 Jul 2021 23:10:04 +0700 Sat, 24 Apr 2021 09:04:18 +0700 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Thu, 01 Jul 2021 23:10:04 +0700 Thu, 01 Jul 2021 22:37:34 +0700 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:
InternalIP: 10.100.10.4
Hostname: ix-truenas
Capacity:
cpu: 32
ephemeral-storage: 1861239040Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 131781728Ki
pods: 110
Allocatable:
cpu: 32
ephemeral-storage: 1810613336692
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 131781728Ki
pods: 110
System Info:
Machine ID: 37f8fb817c7b463ba06d90e2e41c4d9d
System UUID: 00000000-0000-0000-0000-d05099dd378f
Boot ID: 033d0686-ab64-4f21-8f52-364a0a5d7000
Kernel Version: 5.10.42+truenas
OS Image: Debian GNU/Linux 11 (bullseye)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://20.10.6
Kubelet Version: v1.21.0-k3s1
Kube-Proxy Version: v1.21.0-k3s1
PodCIDR: 172.16.0.0/16
PodCIDRs: 172.16.0.0/16
Non-terminated Pods: (1 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- —
ix-chia1 chia1-ix-chart-997d88d9-tnkth 0 (0%) 0 (0%) 0 (0%) 0 (0%) 7d9h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 0 (0%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events:
Type Reason Age From Message

------ ---- ---- -------
Normal Starting 35m kubelet Starting kubelet.
Normal NodeAllocatableEnforced 35m kubelet Updated Node Allocatable limit across pods
Normal NodeHasSufficientMemory 34m kubelet Node ix-truenas status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 34m kubelet Node ix-truenas status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 34m kubelet Node ix-truenas status is now: NodeHasSufficientPID
Warning Rebooted 34m kubelet Node ix-truenas has been rebooted, boot id: 033d0686-ab64-4f21-8f52-364a0a5d7000
Normal NodeNotReady 34m kubelet Node ix-truenas status is now: NodeNotReady

K3s took too long to initialise which resulted in post start call to be missed. However when we tried to reproduce it we were not able to and it works as desired. please let us know if this happens again. Thank you

i'll be available for the next 4 hours, please email me teamviewer credentials if you are free ( the next 4 hours should be in between 0700-2200 hours GMT + 7 ). My email id is waqar@ixsystems.com, looking forward to hearing from you. Thank you

 I am GMT +7 (Bangkok, Thailand), I would like to get this fixed so I can be ready any tome between my 7am and 10-11pm. 

I have reestablished my teamviewer account. This TrueNAS machine is running several storage pools for my homelab and I need to avoid reboots/shutting it down. Several network service VM are being run off the storage share.

I'll stand by for a time that works, thanks!

Derek