Description
Steps to Reproduce
Expected Result
Actual Result
Environment
Hardware Health
Error Message (if applicable)
Attachments
- 14 Jan 2023, 11:36 AM
Activity
Automation for Jira January 19, 2023 at 2:49 PM
This issue has now been closed. Comments made after this point may not be viewed by the TrueNAS Teams. Please open a new issue if you have found a problem or need to re-engage with the TrueNAS Engineering Teams.
Waqar Ahmed January 19, 2023 at 2:49 PM
Marking it as duplicate as this is not a problem in Cobia - for Bluefin it should be marked as third party to resolve imho.
Waqar Ahmed January 19, 2023 at 2:48 PM
The problem is that when kubernetes pulls image with the help of docker using RPC calls, it has an internal time out of 5 minutes iirc.
So if some image takes more then that, it will fail to pull that image unfortunately. We investigated to see what we can do about it but as far as i recall, the changes would be too invasive to manage and is essentially an upstream issue. However this works in next major release i.e Cobia where with containerd ( docker has been removed there ), the issue is non-existent.
Matt January 19, 2023 at 11:13 AM
My internal network speed is good, cat 7 ethernet and 1GB switches, my internet speed however is very low about 8mbs down and 4mbps up , the storage is 4 HDDs in raid z1 so not the fastest but enough for my means.
I ran the pull command and it successfully ran.
I was then able to launch plex, would you mind explaining why this is the case just encase this reoccurs down the line. Was it just timing before downloading the app contents?
Waqar Ahmed January 19, 2023 at 7:39 AM
@Matt can you confirm please if you are able to download the image by running the following command:
docker pull plexinc/pms-docker:1.30.1.6562-915986d62
And then stop/start the app again to confirm it works ?
Also curious what internet speed does your network has ?
Before updating to Bluefin I had 4 apps setup, Plex, Piehole, qbittorrent, and webNUT. Since updating though I have long managed to get qbittorrent and webNUT working by disabling host path authentication. I have not been able to get Plex or Piehole working, Piehole is a truecharts app but the plex install is official and I believe it it likely caused by the same issue. I get the following event long when trying to launch plex.
2023-01-14 11:21:15
0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
2023-01-14 11:21:15
0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.
2023-01-14 11:20:54
Failed to pull image "plexinc/pms-docker:1.30.1.6562-915986d62": rpc error: code = Unknown desc = context canceled
2023-01-14 11:19:31
Back-off pulling image "plexinc/pms-docker:1.30.1.6562-915986d62"
2023-01-14 11:19:31
Error: ImagePullBackOff
2023-01-14 11:19:30
Failed to pull image "plexinc/pms-docker:1.30.1.6562-915986d62": rpc error: code = Unknown desc = context deadline exceeded
2023-01-14 11:19:30
Error: ErrImagePull
2023-01-14 11:17:31
Scaled up replica set plex-6c78bc9cbf to 1
2023-01-14 11:17:31
Successfully assigned ix-plex/plex-6c78bc9cbf-h8z4r to ix-truenas
2023-01-14 11:17:31
Created pod: plex-6c78bc9cbf-h8z4r
2023-01-14 11:17:31
Pulling image "plexinc/pms-docker:1.30.1.6562-915986d62"
I have recreated the apps trying to change settings around and a good complement of reboots but neither app has come back since the bluefin update. I have also attached by kubernetes settings to this ticket.
If you have any ideas on what I can try please let me know as I really need to resolve this before improving an adding new services.