Apps service fails to start reliably; fails with error "Docker service could not be started", but apps are running

Description

I’ve been having an issue with the Apps service in TrueNAS ElectricEel-24.10.2, where upon reboot the app service fails to start; it will report a failure on the “Apps” tab of the dashboard, with an error “Docker service could not be started”. However this seems to just be a UI issue as all of the docker applications have actually been started successfully. This can be verified by going into the shell and running docker ps, which shows al of the services as being up—and of course the obvious way, which is to access the services and see that they’re running just fine.

Once in this state though, the apps cannot be managed, and I cannot find a way to forcibly restart the service in such a way that the UI picks it up. The only method to resolve this (that I’ve found) is to reboot and hope that this time it’ll work.

This seemed to start happening as I put more services onto the system: I wonder if the issue is that it’s waiting for all services to start, and that happens too slowly for it, so it times out. I’ve found that it’s somewhat more reliable if I reduce the load on the system at startup by having a few Virtual Machines that I run not boot at startup, and just booting them manually. This appears to make things work more reliably, which is what led me to believe it’s about the time taken for things to start.

Note that this is me re-opening as the issue still occurs; that one got closed because I failed to respond quickly enough; sorry about that. I’ve got debug logs this time during the time the error is actually occurring; previously I’d avoided rebooting in order to not have the problem reoccur, because when it does the only solution I’ve found is to attempt to reboot several times in a row until the problem doesn’t occur, which is quite miserable.

Problem/Justification

None

Impact

None

Activity

Show:

Bug Clerk February 10, 2025 at 7:00 PM

This issue has now been closed. Comments made after this point may not be viewed by the TrueNAS Teams. Please open a new issue if you have found a problem or need to re-engage with the TrueNAS Engineering Teams.

Bug Clerk February 10, 2025 at 7:00 PM

Thank you for reporting this issue! This has already been reported (see the "Duplicate" ticket linked above), so this issue is being closed to focus effort in the existing ticket.

Bonnie Follweiler February 10, 2025 at 6:12 PM

Good Morning .

I have moved this ticket into our queue to review.

An engineering representative will update with any further questions or details in the near future.

Santosh Ganti February 10, 2025 at 9:58 AM

There is definitely a bug with apps restoring after reboots, the unset mechanism doesn’t properly function in some situations. The docker services fails but containerd shows containers running, this seems to cause some weird timeout with docker hub and after restore I am locked out of updating or starting applications because I believe in some cases a start/restart can trigger an image update check and you are locked out of using apps.

Nathan Wittstock February 8, 2025 at 8:47 PM

By the way: I’m able to get things back into a decent state following the advice in this thread on the community forums: selecting “Unset Pool” and then re-choosing the pool will start showing all of my apps again; the apps themselves seem unaffected and stay running the whole time. This is a little scary because the “Unset Pool” doesn’t have any documentation that I have been able to find, so I wasn’t sure what the outcome would be and had to trust that the folks on the forum would’ve said something if it went badly. It worked though, at least to get it back into a working state. It still fails after rebooting the system though and needs that procedure done again.

Duplicate

Details

Assignee

Reporter

Labels

Components

Fix versions

Priority

More fields

Katalon Platform

Created February 8, 2025 at 8:14 PM
Updated February 10, 2025 at 7:00 PM
Resolved February 10, 2025 at 7:00 PM