K8S-Gateway fails as upstream DNS resolver
Description
Problem/Justification
None
Impact
None
Activity
Bonnie Follweiler December 1, 2021 at 4:55 PM
He will file another ticket if the behavior happens again.
Steven McElligott November 30, 2021 at 2:55 PM
I had to reinstall TrueNAS since I hosed my install before I saved my debug, you can mark this as closed and if it happens again I'll file another report/post debug first!
Bonnie Follweiler November 22, 2021 at 6:07 PM
Thank you for the report, @Steven McElligott .
Can you please attach a debug file to this ticket? To generate a debug file on TrueNAS SCALE, log in to the TrueNAS web interface, go to System Settings > Advanced, then click Save Debug and wait for the file to download to your local system. In TrueNAS SCALE, this option is in System Settings > Advanced.
Cannot Reproduce
Pinned fields
Click on the next to a field label to start pinning.
Details
Details
Assignee
Triage TeamReporter
Steven McElligottImpact
Low
Components
Fix versions
Affects versions
Priority
LowMore fields
Time tracking
More fields
Time trackingKatalon Platform
Linked Test Cases, Katalon Defect Results, Katalon Studio Test Results
Katalon Platform
Linked Test Cases, Katalon Defect Results, Katalon Studio Test Results
Created November 21, 2021 at 10:10 PM
Updated July 6, 2022 at 9:01 PM
Resolved December 1, 2021 at 4:55 PM
TrueNAS-SCALE-22.02-RC.1-1
App Version
k8s-gateway
0.1.8_4.0.11
Application Events
Application Logs
2021-11-21T21:24:59.479045236Z [INFO] 172.16.18.222:60373 - 13166 "A IN www.facebook.com . udp 34 false 512" NOERROR - 0 2.000451506s
2021-11-21T21:24:59.479091969Z [ERROR] plugin/errors: 2 www.facebook.com . A: read udp 172.16.18.225:55262->1.1.1.1:53: i/o timeout
2021-11-21T21:25:00.475911790Z [INFO] 172.16.18.222:60373 - 13166 "A IN www.facebook.com . udp 34 false 512" NOERROR - 0 2.000735175s
2021-11-21T21:25:00.475983547Z [ERROR] plugin/errors: 2 www.facebook.com . A: read udp 172.16.18.225:44423->1.1.1.1:53: i/o timeout
2021-11-21T21:25:00.476015184Z [INFO] 172.16.18.222:60373 - 13166 "A IN www.facebook.com . udp 34 false 512" NOERROR - 0 2.000778594s
2021-11-21T21:25:00.476031460Z [ERROR] plugin/errors: 2 www.facebook.com . A: read udp 172.16.18.225:58343->1.1.1.1:53: i/o timeout
2021-11-21T21:25:01.479723896Z [INFO] 172.16.18.222:60373 - 13166 "A IN www.facebook.com . udp 34 false 512" NOERROR - 0 2.000650755s
2021-11-21T21:25:01.479800807Z [ERROR] plugin/errors: 2 www.facebook.com . A: read udp 172.16.18.225:47750->1.0.0.1:53: i/o timeout
2021-11-21T21:25:01.479833390Z [INFO] 172.16.18.222:60373 - 13166 "A IN www.facebook.com . udp 34 false 512" NOERROR - 0 2.000733604s
2021-11-21T21:25:01.479850100Z [ERROR] plugin/errors: 2 www.facebook.com . A: read udp 172.16.18.225:33371->1.0.0.1:53: i/o timeout
2021-11-21T21:25:05.243402350Z [INFO] 172.16.18.222:49286 - 7958 "A IN gateway.facebook.com. udp 38 false 512" NOERROR - 0 2.000467543s
2021-11-21T21:25:05.243594151Z [ERROR] plugin/errors: 2 gateway.facebook.com. A: read udp 172.16.18.225:38697->1.1.1.1:53: i/o timeout
2021-11-21T21:25:06.249426602Z [INFO] 172.16.18.222:49286 - 7958 "A IN gateway.facebook.com. udp 38 false 512" NOERROR - 0 2.001261538s
2021-11-21T21:25:06.249513095Z [INFO] 172.16.18.222:49286 - 7958 "A IN gateway.facebook.com. udp 38 false 512" NOERROR - 0 2.00137935s
2021-11-21T21:25:06.249556799Z [ERROR] plugin/errors: 2 gateway.facebook.com. A: read udp 172.16.18.225:39882->1.0.0.1:53: i/o timeout
2021-11-21T21:25:06.249597329Z [ERROR] plugin/errors: 2 gateway.facebook.com. A: read udp 172.16.18.225:52838->1.0.0.1:53: i/o timeout
2021-11-21T21:25:07.906013877Z [INFO] 172.16.18.222:36856 - 9451 "A IN www.facebook.com . udp 34 false 512" NOERROR - 0 2.000738864s
2021-11-21T21:25:07.906058254Z [ERROR] plugin/errors: 2 www.facebook.com . A: read udp 172.16.18.225:48083->1.1.1.1:53: i/o timeout
2021-11-21T21:25:08.908005434Z [INFO] 172.16.18.222:36856 - 9451 "A IN www.facebook.com . udp 34 false 512" NOERROR - 0 2.000213808s
2021-11-21T21:25:08.908058446Z [ERROR] plugin/errors: 2 www.facebook.com . A: read udp 172.16.18.225:46357->1.0.0.1:53: i/o timeout
2021-11-21T21:25:08.908082619Z [INFO] 172.16.18.222:36856 - 9451 "A IN www.facebook.com . udp 34 false 512" NOERROR - 0 2.000214036s
2021-11-21T21:25:08.908115077Z [ERROR] plugin/errors: 2 www.facebook.com . A: read udp 172.16.18.225:46949->1.1.1.1:53: i/o timeout
2021-11-21T21:25:09.908969758Z [INFO] 172.16.18.222:50126 - 27268 "A IN www.facebook.com . udp 34 false 512" NOERROR - 0 2.000878865s
2021-11-21T21:25:09.909032915Z [ERROR] plugin/errors: 2 www.facebook.com . A: read udp 172.16.18.225:58089->1.0.0.1:53: i/o timeout
2021-11-21T21:25:11.910311740Z [INFO] 172.16.18.222:50126 - 27268 "A IN www.facebook.com . udp 34 false 512" NOERROR - 0 2.000899409s
2021-11-21T21:25:11.910516310Z [ERROR] plugin/errors: 2 www.facebook.com . A: read udp 172.16.18.225:53563->1.1.1.1:53: i/o timeout
2021-11-21T21:25:11.910601223Z [INFO] 172.16.18.222:50126 - 27268 "A IN www.facebook.com . udp 34 false 512" NOERROR - 0 2.000944164s
2021-11-21T21:25:11.910709464Z [ERROR] plugin/errors: 2 www.facebook.com . A: read udp 172.16.18.225:41607->1.0.0.1:53: i/o timeout
2021-11-21T21:25:13.936272987Z [INFO] 172.16.18.222:54237 - 7465 "A IN gateway.facebook.com. udp 38 false 512" NOERROR - 0 2.000725327s
2021-11-21T21:25:13.936459370Z [ERROR] plugin/errors: 2 gateway.facebook.com. A: read udp 172.16.18.225:53298->1.1.1.1:53: i/o timeout
2021-11-21T21:25:14.942384804Z [INFO] 172.16.18.222:54237 - 7465 "A IN gateway.facebook.com. udp 38 false 512" NOERROR - 0 2.00109242s
2021-11-21T21:25:14.942575303Z [INFO] 172.16.18.222:54237 - 7465 "A IN gateway.facebook.com. udp 38 false 512" NOERROR - 0 2.000996329s
2021-11-21T21:25:14.942717028Z [ERROR] plugin/errors: 2 gateway.facebook.com. A: read udp 172.16.18.225:59410->1.0.0.1:53: i/o timeout
2021-11-21T21:25:14.942812379Z [ERROR] plugin/errors: 2 gateway.facebook.com. A: read udp 172.16.18.225:58352->1.0.0.1:53: i/o timeout
2021-11-21T21:25:15.941170863Z [INFO] 172.16.18.222:46825 - 38477 "A IN gateway.facebook.com. udp 38 false 512" NOERROR - 0 2.000553195s
2021-11-21T21:25:15.941333295Z [ERROR] plugin/errors: 2 gateway.facebook.com. A: read udp 172.16.18.225:53195->1.1.1.1:53: i/o timeout
2021-11-21T21:25:16.327981291Z [INFO] 172.16.18.222:49192 - 6505 "A IN www.facebook.com . udp 34 false 512" NOERROR - 0 2.001187738s
2021-11-21T21:25:16.328146836Z [ERROR] plugin/errors: 2 www.facebook.com . A: read udp 172.16.18.225:46113->1.1.1.1:53: i/o timeout
2021-11-21T21:25:17.337871017Z [INFO] 172.16.18.222:49192 - 6505 "A IN www.facebook.com . udp 34 false 512" NOERROR - 0 2.000813575s
2021-11-21T21:25:17.338050495Z [INFO] 172.16.18.222:49192 - 6505 "A IN www.facebook.com . udp 34 false 512" NOERROR - 0 2.000859141s
2021-11-21T21:25:17.338138705Z [ERROR] plugin/errors: 2 www.facebook.com . A: read udp 172.16.18.225:54316->1.0.0.1:53: i/o timeout
2021-11-21T21:25:17.338220739Z [ERROR] plugin/errors: 2 www.facebook.com . A: read udp 172.16.18.225:55548->1.0.0.1:53: i/o timeout
2021-11-21T21:25:17.893020468Z [INFO] 172.16.18.222:42311 - 40953 "A IN forums.redflagdeals.com. udp 41 false 512" NOERROR - 0 2.000765325s
2021-11-21T21:25:17.893070270Z [ERROR] plugin/errors: 2 forums.redflagdeals.com. A: read udp 172.16.18.225:54644->1.0.0.1:53: i/o timeout
2021-11-21T21:25:17.893093567Z [INFO] 172.16.18.222:54668 - 8454 "A IN assets.rfdcontent.com. udp 39 false 512" NOERROR - 0 2.000466518s
2021-11-21T21:25:17.893123537Z [ERROR] plugin/errors: 2 assets.rfdcontent.com. A: read udp 172.16.18.225:34773->1.0.0.1:53: i/o timeout
2021-11-21T21:25:17.942552948Z [INFO] 172.16.18.222:46825 - 38477 "A IN gateway.facebook.com. udp 38 false 512" NOERROR - 0 2.001045208s
2021-11-21T21:25:17.942599103Z [ERROR] plugin/errors: 2 gateway.facebook.com. A: read udp 172.16.18.225:53378->1.0.0.1:53: i/o timeout
2021-11-21T21:25:17.942629540Z [INFO] 172.16.18.222:46825 - 38477 "A IN gateway.facebook.com. udp 38 false 512" NOERROR - 0 2.000964241s
2021-11-21T21:25:17.942655848Z [ERROR] plugin/errors: 2 gateway.facebook.com. A: read udp 172.16.18.225:37311->1.1.1.1:53: i/o timeout
2021-11-21T21:25:18.898109877Z [INFO] 172.16.18.222:42311 - 40953 "A IN forums.redflagdeals.com. udp 41 false 512" NOERROR - 0 2.000614995s
2021-11-21T21:25:18.898156988Z [INFO] 172.16.18.222:54668 - 8454 "A IN assets.rfdcontent.com. udp 39 false 512" NOERROR - 0 2.000380591s
2021-11-21T21:25:18.898176423Z [ERROR] plugin/errors: 2 forums.redflagdeals.com. A: read udp 172.16.18.225:46581->1.0.0.1:53: i/o timeout
2021-11-21T21:25:18.898193436Z [ERROR] plugin/errors: 2 assets.rfdcontent.com. A: read udp 172.16.18.225:41410->1.1.1.1:53: i/o timeout
2021-11-21T21:25:18.898210358Z [INFO] 172.16.18.222:42311 - 40953 "A IN forums.redflagdeals.com. udp 41 false 512" NOERROR - 0 2.000786474s
2021-11-21T21:25:18.898226873Z [ERROR] plugin/errors: 2 forums.redflagdeals.com. A: read udp 172.16.18.225:43994->1.1.1.1:53: i/o timeout
2021-11-21T21:25:18.899249856Z [INFO] 172.16.18.222:54668 - 8454 "A IN assets.rfdcontent.com. udp 39 false 512" NOERROR - 0 2.001345302s
2021-11-21T21:25:18.899277648Z [ERROR] plugin/errors: 2 assets.rfdcontent.com. A: read udp 172.16.18.225:51094->1.1.1.1:53: i/o timeout
2021-11-21T21:25:19.901894271Z [INFO] 172.16.18.222:58865 - 943 "A IN netdna.bootstrapcdn.com. udp 41 false 512" NOERROR - 0 2.001034252s
2021-11-21T21:25:19.901949946Z [INFO] 172.16.18.222:50813 - 21680 "A IN ajax.googleapis.com. udp 37 false 512" NOERROR - 0 2.000952269s
2021-11-21T21:25:19.901987101Z [ERROR] plugin/errors: 2 netdna.bootstrapcdn.com. A: read udp 172.16.18.225:43211->1.0.0.1:53: i/o timeout
2021-11-21T21:25:19.902024846Z [INFO] 172.16.18.222:43331 - 57097 "A IN fonts.googleapis.com. udp 38 false 512" NOERROR - 0 2.000920886s
2021-11-21T21:25:19.902059685Z [ERROR] plugin/errors: 2 ajax.googleapis.com. A: read udp 172.16.18.225:43249->1.0.0.1:53: i/o timeout
2021-11-21T21:25:19.902082317Z [ERROR] plugin/errors: 2 fonts.googleapis.com. A: read udp 172.16.18.225:42250->1.0.0.1:53: i/o timeout
2021-11-21T21:25:19.911212624Z [INFO] 172.16.18.222:57580 - 12026 "A IN google.com. udp 28 false 512" NOERROR - 0 2.000888385s
2021-11-21T21:25:19.911249563Z [ERROR] plugin/errors: 2 google.com. A: read udp 172.16.18.225:51376->1.1.1.1:53: i/o timeout
2021-11-21T21:25:20.439796894Z [INFO] 172.16.18.222:40282 - 41833 "A IN gateway.facebook.com. udp 38 false 512" NOERROR - 0 2.00055445s
Application Configuration
k8s-gateway
cancel
Application Name *
k8s-gateway
Controller
Show Advanced Controller Settings
Show Expert Configuration Options
Container Configuration
Image Environment
Timezone
'America/Toronto' timezone
UMASK
help_outline
002
Configure Image environment
No items have been added yet.
Show Expert Config
App Configuration
Configure Domains
Domain name *
MYDOMAIN.COM <-------------- Changed
Forward dnsChallenge
help_outline
Enable
Forward DNS To
Enabled
Primary *
1.1.1.1
Secondary *
1.0.0.1
Configure Forward Options
No items have been added yet.
Networking and Services
Configure Service(s)
Main Service
help_outline
Service Type
help_outline
Simple
Service's Port(s) Configuration
Main Service Port Configuration
Port *
help_outline
5353 <-------------------------Changed
Show Advanced settings
Show Expert Config
Storage and Persistence
Configure Additional app storage
No items have been added yet.
Security and Permissions
Show Advanced Security Settings
Pod Security Context
runAsUser
help_outline
0
runAsGroup
help_outline
0
fsGroup
help_outline
568
Configure supplemental Groups
No items have been added yet.
When should we take ownership?
OnRootMismatch
Resources and Devices
Set Custom Resource Limits/Requests (Advanced)
Configure Mount USB devices
No items have been added yet.
GPU Configuration
GPU Resource (nvidia.com/gpu)
help_outline
Allocate 0 nvidia.com/gpu GPU
Addons
VPN
Type
disabled
Codeserver
enabled
Promtail
enabled
Netshoot
enabled
Advanced
(Advanced) Horizontal Pod Autoscaler
enabled
Describe the bug
DNS queries aren't being resolved when using PiHole and k8s-gateway.
PiHole seems to be sending the DNS queries upstream (since there's requests inside k8s-gateway) and resolves internal queries but times out with external queries using default DNS
To Reproduce
Install pihole, use truenasip:5353 as upstream DNS
Install k8s-gateway, use 5353 as udp ports since 53 is used by Pi-Hole, use 1.1.1.1 and 1.0.0.1 (the defaults for k8s-gateway) as DNS servers
External queries aren't resolved
Expected behavior
Resolv
Screenshots
PiHole DNS screen:
Additional context
If I use Local DNS inside PiHole it resolves fine so I dunno why k8s-gateway doesn't like me lol