pool import taking > 15mins

Description

Upgraded my system from Cobia 23.10.2 to Dragonfish this morning, and the apps service won't start. I suspect this is related to the fact that `ix-zfs.service` timed out after 15 minutes on boot, importing only the main data pool but not my apps pool (though that did later import without any intervention from me). After waiting a half hour or so in hopes the system would fix itself (with no change), I rebooted. Again, ix-zfs.service timed out after 15 minutes, and my apps pool was not imported when the console menu came up, though it was a few minutes later. Currently 45 minutes' uptime and the apps service still isn't running.

Host ID: 7d1eea8f11b3b2406dfd6c134c29ca471f5f04f6bfeeefd5a8816c49a7377db3

Session ID: 4d2ef7fd-d5d5-b8b2-7827-b16535a084e9

Steps to Reproduce

None

Expected Result

None

Actual Result

None

Environment

None

Hardware Health

None

Error Message (if applicable)

Attachments

10
  • 22 May 2024, 11:19 AM
  • 22 May 2024, 11:17 AM
  • 22 May 2024, 11:17 AM
  • 22 May 2024, 11:17 AM
  • 22 May 2024, 11:17 AM
  • 22 May 2024, 11:17 AM
  • 22 May 2024, 11:17 AM
  • 22 May 2024, 11:17 AM
  • 22 May 2024, 11:17 AM
  • 22 May 2024, 11:17 AM

Activity

Show:

Morgan Littlewood June 6, 2024 at 6:57 PM

Dan Brown June 6, 2024 at 6:38 PM

If memory serves, I migrated from CORE to Bluefin, so shortly after Bluefin’s release. I’m afraid I don’t remember more specifically.

Over 50k of the snapshots were coming from my “pull” replication process from my Proxmox hosts. Snapshots are expiring just fine on those hosts, but the NAS isn’t expiring them. This clearly warrants more digging, but at least I know roughly where to dig.

Morgan Littlewood June 6, 2024 at 4:30 PM

When was this SCALE system installed? In the early versions, we had issue with SCALE snapshots and Apps until we supported dockerfs.

Dan Brown June 6, 2024 at 9:12 AM

I need to figure out where all these snapshots are coming from. After spending some time pruning them (even with scripting, it was over 24 hours), I’m down from over 100k snapshots on the system to just over 10k. Ran the update to 24.0.1.1, and the pool import on boot completes without a problem, the system continues to boot, the apps come up, all as expected. Thanks for the pointer.

ix-zfs still took longer to finish than under Cobia (52 seconds under Cobia; 98 seconds under Dragonfish), but obviously far better than 20 minutes.

Caleb June 4, 2024 at 2:34 PM

Nope, no need to remove. I’m still investigating what can be done to work-around it in the meantime. I’ll keep you updated.

Duplicate
Pinned fields
Click on the next to a field label to start pinning.

Assignee

Caleb

Reporter

Fix versions

Components

Priority

Created April 26, 2024 at 10:46 AM
Updated June 6, 2024 at 6:57 PM
Resolved May 30, 2024 at 12:27 AM