Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Thanks for using the TrueNAS Community Edition issue tracker! TrueNAS Enterprise users receive direct support for their reports from our support portal.

k3s fails (Random database disk image is malformed errors)

Description

Im running TrueNAS-SCALE-22.12.2 and getting database disk image is malformed errors.

if checked multipe things related to the k3s cluster and if trown away the ix-applcation “folder“ in the used data pool ( rebuilded the cluster). below the errors. below that the smart data of the nvme disk where the cluster is running on

May 23 13:18:20 truenas k3s[47257]: time="2023-05-23T13:18:20+02:00" level=error msg="error while range on /registry/podtemplates/ /registry/podtemplates/: database disk image is malformed" May 23 13:18:20 truenas k3s[47257]: time="2023-05-23T13:18:20+02:00" level=error msg="error while range on /registry/events/ /registry/events/: database disk image is malformed" May 23 13:18:20 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:20.128+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002cea000/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:20 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:20.129+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00383e540/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:20 truenas k3s[47257]: W0523 13:18:20.134406 47257 genericapiserver.go:656] Skipping API storage.k8s.io/v1alpha1 because it has no resources. May 23 13:18:20 truenas k3s[47257]: W0523 13:18:20.153130 47257 genericapiserver.go:656] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources. May 23 13:18:20 truenas k3s[47257]: time="2023-05-23T13:18:20+02:00" level=error msg="error while range on /registry/apiextensions.k8s.io/customresourcedefinitions/ /registry/apiextensions.k8s.io/customresourcedefinitions/: database disk image is malformed" May 23 13:18:20 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:20.159+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000c52540/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:20 truenas k3s[47257]: W0523 13:18:20.182752 47257 genericapiserver.go:656] Skipping API apps/v1beta2 because it has no resources. May 23 13:18:20 truenas k3s[47257]: W0523 13:18:20.183321 47257 genericapiserver.go:656] Skipping API apps/v1beta1 because it has no resources. May 23 13:18:20 truenas k3s[47257]: W0523 13:18:20.186499 47257 genericapiserver.go:656] Skipping API admissionregistration.k8s.io/v1beta1 because it has no resources. May 23 13:18:20 truenas k3s[47257]: W0523 13:18:20.213756 47257 genericapiserver.go:656] Skipping API events.k8s.io/v1beta1 because it has no resources. May 23 13:18:20 truenas k3s[47257]: I0523 13:18:20.215293 47257 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStora> May 23 13:18:20 truenas k3s[47257]: I0523 13:18:20.215321 47257 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,> May 23 13:18:20 truenas k3s[47257]: time="2023-05-23T13:18:20+02:00" level=error msg="error while range on /registry/persistentvolumeclaims/ /registry/persistentvolumeclaims/: database disk image is malformed" May 23 13:18:20 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:20.236+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc003636e00/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:20 truenas k3s[47257]: W0523 13:18:20.262304 47257 genericapiserver.go:656] Skipping API apiregistration.k8s.io/v1beta1 because it has no resources. May 23 13:18:20 truenas k3s[47257]: time="2023-05-23T13:18:20+02:00" level=error msg="error while range on /registry/minions/ /registry/minions/: database disk image is malformed" May 23 13:18:20 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:20.317+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0039fae00/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:20 truenas k3s[47257]: time="2023-05-23T13:18:20+02:00" level=error msg="error while range on /registry/namespaces/ /registry/namespaces/: database disk image is malformed" May 23 13:18:20 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:20.391+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002ceafc0/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:20 truenas k3s[47257]: time="2023-05-23T13:18:20+02:00" level=error msg="error while range on /registry/resourcequotas/ /registry/resourcequotas/: database disk image is malformed" May 23 13:18:20 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:20.405+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00383f500/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:20 truenas k3s[47257]: time="2023-05-23T13:18:20+02:00" level=error msg="error while range on /registry/serviceaccounts/ /registry/serviceaccounts/: database disk image is malformed" May 23 13:18:20 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:20.483+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0041fce00/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:20 truenas k3s[47257]: time="2023-05-23T13:18:20+02:00" level=error msg="error while range on /registry/configmaps/ /registry/configmaps/: database disk image is malformed" May 23 13:18:20 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:20.533+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0009f0e00/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:20 truenas k3s[47257]: time="2023-05-23T13:18:20+02:00" level=error msg="error while range on /registry/secrets/ /registry/secrets/: database disk image is malformed" May 23 13:18:20 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:20.638+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000c53340/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:20 truenas k3s[47257]: time="2023-05-23T13:18:20+02:00" level=error msg="error while range on /registry/services/endpoints/ /registry/services/endpoints/: database disk image is malformed" May 23 13:18:20 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:20.649+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00407ac40/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:20 truenas k3s[47257]: time="2023-05-23T13:18:20+02:00" level=error msg="error while range on /registry/services/specs/ /registry/services/specs/: database disk image is malformed" May 23 13:18:20 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:20.714+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001830380/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:20 truenas k3s[47257]: time="2023-05-23T13:18:20+02:00" level=error msg="error while range on /registry/controllers/ /registry/controllers/: database disk image is malformed" May 23 13:18:20 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:20.819+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00181d340/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:20 truenas k3s[47257]: time="2023-05-23T13:18:20+02:00" level=error msg="error while range on /registry/cronjobs/ /registry/cronjobs/: database disk image is malformed" May 23 13:18:20 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:20.879+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc003582e00/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:21 truenas k3s[47257]: time="2023-05-23T13:18:21+02:00" level=error msg="error while range on /registry/persistentvolumes/ /registry/persistentvolumes/: database disk image is malformed" May 23 13:18:21 truenas k3s[47257]: time="2023-05-23T13:18:21+02:00" level=error msg="error while range on /registry/limitranges/ /registry/limitranges/: database disk image is malformed" May 23 13:18:21 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:21.029+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc003e4efc0/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:21 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:21.029+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc003e4e000/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:21 truenas k3s[47257]: time="2023-05-23T13:18:21+02:00" level=error msg="error while range on /registry/pods/ /registry/pods/: database disk image is malformed" May 23 13:18:21 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:21.039+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00181c380/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:21 truenas k3s[47257]: time="2023-05-23T13:18:21+02:00" level=error msg="error while range on /registry/horizontalpodautoscalers/ /registry/horizontalpodautoscalers/: database disk image is malformed" May 23 13:18:21 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:21.236+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002ceb340/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:21 truenas k3s[47257]: time="2023-05-23T13:18:21+02:00" level=error msg="error while range on /registry/horizontalpodautoscalers/ /registry/horizontalpodautoscalers/: database disk image is malformed" May 23 13:18:21 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:21.317+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc003ddc000/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:21 truenas k3s[47257]: time="2023-05-23T13:18:21+02:00" level=error msg="error while range on /registry/jobs/ /registry/jobs/: database disk image is malformed" May 23 13:18:21 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:21.618+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002f17dc0/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:21 truenas k3s[47257]: time="2023-05-23T13:18:21+02:00" level=error msg="error while range on /registry/networkpolicies/ /registry/networkpolicies/: database disk image is malformed" May 23 13:18:21 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:21.905+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00377e380/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:21 truenas k3s[47257]: time="2023-05-23T13:18:21+02:00" level=error msg="error while range on /registry/clusterroles/ /registry/clusterroles/: database disk image is malformed" May 23 13:18:21 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:21.936+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc003072000/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:21 truenas k3s[47257]: time="2023-05-23T13:18:21+02:00" level=error msg="error while range on /registry/validatingwebhookconfigurations/ /registry/validatingwebhookconfigurations/: database disk image is malformed" May 23 13:18:21 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:21.972+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc003daf340/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:22 truenas k3s[47257]: time="2023-05-23T13:18:22+02:00" level=error msg="error while range on /registry/flowschemas/ /registry/flowschemas/: database disk image is malformed" May 23 13:18:22 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:22.231+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc004909dc0/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:22 truenas k3s[47257]: time="2023-05-23T13:18:22+02:00" level=error msg="error while range on /registry/ingressclasses/ /registry/ingressclasses/: database disk image is malformed" May 23 13:18:22 truenas k3s[47257]: {"level":"warn","ts":"2023-05-23T13:18:22.240+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc003c94e00/kine.sock","attempt":0,"error":"rpc error: code = > May 23 13:18:22 truenas k3s[47257]: time="2023-05-23T13:18:22+02:00" level=error msg="error while range on /registry/clusterrolebindings/ /registry/clusterrolebindings/: database disk image is malformed" lines 57-143

 

smartctl --all /dev/nvme0n1 smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.79+truenas] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Number: Samsung SSD 980 PRO 1TB Serial Number: S5GXNF0T309254W Firmware Version: 5B2QGXA7 PCI Vendor/Subsystem ID: 0x144d IEEE OUI Identifier: 0x002538 Total NVM Capacity: 1,000,204,886,016 [1.00 TB] Unallocated NVM Capacity: 0 Controller ID: 6 NVMe Version: 1.3 Number of Namespaces: 1 Namespace 1 Size/Capacity: 1,000,204,886,016 [1.00 TB] Namespace 1 Utilization: 628,009,619,456 [628 GB] Namespace 1 Formatted LBA Size: 512 Namespace 1 IEEE EUI-64: 002538 b321b8937a Local Time is: Tue May 23 13:30:26 2023 CEST Firmware Updates (0x16): 3 Slots, no Reset required Optional Admin Commands (0x0017): Security Format Frmw_DL Self_Test Optional NVM Commands (0x0057): Comp Wr_Unc DS_Mngmt Sav/Sel_Feat Timestmp Log Page Attributes (0x0f): S/H_per_NS Cmd_Eff_Lg Ext_Get_Lg Telmtry_Lg Maximum Data Transfer Size: 128 Pages Warning Comp. Temp. Threshold: 82 Celsius Critical Comp. Temp. Threshold: 85 Celsius Supported Power States St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat 0 + 8.49W - - 0 0 0 0 0 0 1 + 4.48W - - 1 1 1 1 0 200 2 + 3.18W - - 2 2 2 2 0 1000 3 - 0.0400W - - 3 3 3 3 2000 1200 4 - 0.0050W - - 4 4 4 4 500 9500 Supported LBA Sizes (NSID 0x1) Id Fmt Data Metadt Rel_Perf 0 + 512 0 0 === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 32 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 2% Data Units Read: 8,332,509 [4.26 TB] Data Units Written: 32,737,761 [16.7 TB] Host Read Commands: 43,621,490 Host Write Commands: 856,066,928 Controller Busy Time: 6,220 Power Cycles: 25 Power On Hours: 1,035 Unsafe Shutdowns: 18 Media and Data Integrity Errors: 0 Error Information Log Entries: 0 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 1: 32 Celsius Temperature Sensor 2: 34 Celsius Error Information (NVMe Log 0x01, 16 of 64 entries) No Errors Logged

Problem/Justification

None

Impact

None

Activity

Show:

Bonnie Follweiler June 26, 2023 at 2:28 PM

At this time there is insufficient information to proceed with the investigation. If at any time additional debugging information is supplied this ticket may be reopened for evaluation.

Bonnie Follweiler June 16, 2023 at 4:38 PM

Thank you for submitting this ticket .
We will need a debug file from the affected system in order to investigate your issue. Please upload the debug file, and any other information related to this issue, to our secure and private upload service located at https://ixsystems.atlassian.net/servicedesk/customer/portal/15/group/37/create/153 .

Debug files can be generated in the UI by navigating to System Settings -> Advanced -> Save Debug.

Need additional information
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Labels

Components

Fix versions

Affects versions

Priority

More fields

Katalon Platform

Created May 23, 2023 at 11:35 AM
Updated May 8, 2024 at 6:08 PM
Resolved June 26, 2023 at 2:28 PM
Configure