Unable to open jail console if pool is offline

Description

I have two pools: my primary data pool and a backup pool. My backup pool has gone offline and that has blocked my ability to open any iocage console even though the jails don’t make use of the pool at all. When I attempt to start the console I see:

freenas% sudo iocage console plex
Password:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/zfs.py", line 20, in run
    cp.check_returncode()
  File "/usr/local/lib/python3.9/subprocess.py", line 460, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['zfs', 'get', '-H', '-o', 'name,property,value', '-t', 'filesystem', 'all', 'External Backups', 'boot-pool', 'motherlode']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1656, in invoke
    super().invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/iocage_cli/__init__.py", line 259, in cli
    ioc_check.IOCCheck()
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_check.py", line 50, in __init__
    self.pool = iocage_lib.ioc_json.IOCJson(
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_json.py", line 1372, in __init__
    super().__init__(location, checking_datasets, silent, callback)
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_json.py", line 429, in __init__
    self.pool, self.iocroot = self.get_pool_and_iocroot()
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_json.py", line 553, in get_pool_and_iocroot
    pool = get_pool()
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_json.py", line 452, in get_pool
    zpools = [pool for pool in PoolListableResource() if not pool.root_dataset.locked]
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/ioc_json.py", line 452, in <listcomp>
    zpools = [pool for pool in PoolListableResource() if not pool.root_dataset.locked]
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/pools.py", line 74, in root_dataset
    return Dataset(self.name, cache=self.cache)
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/dataset.py", line 38, in __init__
    self._properties = deepcopy(cache.datasets.get(self.resource_name))
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/cache.py", line 72, in datasets
    ioc_pool = self.iocage_activated_pool_internal(lock=False)
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/cache.py", line 42, in iocage_activated_pool_internal
    all_properties([p for p in pools], types=['filesystem'])
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/zfs.py", line 72, in all_properties
    data = run([
  File "/usr/local/lib/python3.9/site-packages/iocage_lib/zfs.py", line 22, in run
    raise ZFSException(cp.returncode, cp.stderr)
iocage_lib.zfs.ZFSException: cannot open 'External Backups': pool I/O is currently suspended

https://ixsystems.atlassian.net/browse/NAS-112979 looks like a potentially related issue, although that was for an older version of TrueNAS.

Problem/Justification

None

Impact

None

Activity

Show:

Bonnie Follweiler September 18, 2023 at 2:01 PM

At this time there is insufficient information to proceed with the investigation. If at any time additional debugging information is supplied this ticket may be reopened for evaluation.

Bonnie Follweiler September 5, 2023 at 1:51 PM

Good Morning @Kevin Menard.

We will need a debug from the affected system to investigate this issue. Please upload the Debug file to our secure and private upload service located at https://ixsystems.atlassian.net/servicedesk/customer/portal/15/group/37/create/153 .

Automation for Jira September 4, 2023 at 3:50 AM

Thank you for submitting this TrueNAS Bug Report! So that we can quickly investigate your issue, please attach a Debug file and any other information related to this issue through our secure and private upload service below. Debug files can be generated in the UI by navigating to System -> Advanced -> Save Debug.

https://ixsystems.atlassian.net/servicedesk/customer/portal/15/group/37/create/153