Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

After updating to Cobia, k3s doesn't start

Description

After updating to Cobia, k3s doesn't start. I've tried unsetting the pool, restarting, then resetting the pool, updating the Kubernetes settings and the network settings, removing any taints from the ix-truenas node, deleting all pods and scaling all deployments down to 0 replicas, etc. with no success. I did notice that upon describing the ix-truenas node, I do get the "cni plugin not initialized" error, and I'm not sure what to do about it. I noticed that whenever the pool is unset then set again, /etc/cni/net.d either gets deleted or its contents are wiped. I also noticed that all of my network interfaces appear to be disabled even though I can access the UI just fine. I'm not sure if any of these things are red herrings, but I've been trying to troubleshoot this issue for several weeks now and could do with some help. Let me know what logs I can upload to help. Thanks!

Problem/Justification

None

Impact

None

Activity

Show:

Waqar September 28, 2023 at 4:46 PM

Thank you for confirming! Glad you have it running nicely.

Jason Katsaros September 27, 2023 at 6:07 PM
Edited

Hello, Waqar! Thank you so much for your help with this issue!

I did as you suggested and switched the pool to an SSD, which resolved this issue. I ended up doing a fresh install of TrueNAS since I was still having Kubernetes networking issues after the switch, but I think that was a me problem. Haha. My apps are back up and running, thanks to your help! This issue can now be closed. Thanks again! I appreciate it!

Waqar September 22, 2023 at 12:06 PM

various errors being pointed out in k3s logs stating the deadline exceeded statement and in the context/order it appears, it likely points to that issue. Currently k3s is not starting at all for you so migrating is tricky for you because we can’t retrieve the context from k3s. Whenever you upgrade, a backup is created automatically for you - that can be restored and then you can select a new pool and click on migrate boolean flag in the UI and it should migrate your existing apps there nicely.
I guess we can setup a time and i can poke around your system a bit too and then help you with the restoration process because that is a bit tricky.

Jason Katsaros September 20, 2023 at 4:10 PM

Hello, Waqar! Thank you very much for getting back to me! I really appreciate it!

Do you mind pointing out where in the logs you noticed that the database was corrupted? I could try investigating that if I know where to start.

Do you know why updating to Cobia would’ve kickstarted this issue? I’ve been using TrueNAS (both CORE and SCALE) for a while now and have never had any problems with my HDDs before (I’ve had the same hardware setup this whole time).

Do you have any guides on how to migrate from HDDs to SSDs for the pool? Do I just need to unset my current pool and set a pool that’s using SSDs and TrueNAS will figure out the rest?

Thank you very much again!

Waqar September 20, 2023 at 3:41 PM

it seems your apps pool is entirely made up of HDDs, we recommend using SSDs for apps because of I/O of apps/k3s on HDDs is not really nice and we see cases of k3s database getting corrupted. Can you please set it up on a SSD pool and see if you experience any problems there ? I am not able to reproduce your problem here locally and the logs it seems to indicate that the database is corrupted ( ).

Cannot Reproduce

Details

Assignee

Reporter

Labels

Ready For Review?

True

Original estimate

Time remaining

0m

Components

Fix versions

Affects versions

Priority

Katalon Platform

Created September 12, 2023 at 11:02 PM
Updated February 27, 2025 at 9:19 PM
Resolved September 28, 2023 at 4:46 PM