Faulted pool - Metadata Corrupted

Description

I was running on 12.0-U1 for a little while until i saw NAS-108627. Upgraded to 12.0-U1.1 last night 1/16/2021 between 6PM - 8PM. Came back up fine. Had a power failure today. when booting systems back up I found out that my main data pool would not import.

state: Faulted
status: The pool metadata is corrupted.

action: The pool cannot be imported due to damaged devices or data.

The pool may be active on another ssytem but can be imported using the '-f' flag.

scrub was in process when system went down.

what I have done:
sudo zpool import -fF -R /mnt vault

waiting for scrub to finish. will start another scrub.

Problem/Justification

None

Impact

None

SmartDraw Connector

Katalon Manual Tests (BETA)

Activity

Show:

William Gryzbowski May 12, 2021 at 12:17 PM

have you experience any of this again?

Alexander Motin January 21, 2021 at 3:55 PM

Corruption after power loss can be the result of not properly working disk cache flushes, in which context pretty rare unknown HBA does not make me feel good.  In its specs I see some mention of "Disk Read Ahead and Write Back cache", whatever it means, though it can be meaningless marketing noise.

Also I see at least two of your HDDs are complaining about pending (which means unreadable) sectors, that may or may not be related, but either way not good.

Or the corruption can be really a corruption, in which case we have nowhere to start rather than blindly try to reproduce it, unless you have some additional information.  During pool import middleware runs additional dtrace script to dump ZFS internal log to the terminal, but I don't think it goes into logs, at least I can't find it in the debug.

Andrew Hurlburt January 21, 2021 at 3:33 PM

2 scrubs came back clean. rebooted and pool is importing on boot. no data issues found yet.

Alexander Motin January 20, 2021 at 7:06 PM

did not cause physical pool corruption, it caused some garbage reads instead of user data in certain situations.

Do I understand right that after you've imported the pool in recovery mode it happily recovered and experience no problems?

Andrew Hurlburt January 19, 2021 at 1:03 AM

Second scrub finished no errors.

Cannot Reproduce
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Fix versions

Affects versions

Priority

More fields

Katalon Platform

Created January 17, 2021 at 10:09 PM
Updated July 1, 2022 at 5:14 PM
Resolved June 28, 2021 at 9:31 PM