Node is unable to rejoin cluster after power off/on of 1/4 nodes
Description
Problem/Justification
None
Impact
None
Activity
Show:
William Gryzbowski August 19, 2022 at 11:39 AM
you will need 22.02.4.
Erik Deridder August 19, 2022 at 10:24 AM
I have kind of the same problem. Successfully created a 3 node cluster all working. At evening shutdown the complete cluster. In the morning started up all nodes and cluster doesn't respond to cluster ip addresses anymore. One node also didn’t start the smb service. Looks kind of the same issue
Bug Clerk August 11, 2022 at 5:53 PM
22.02.4 PR: https://github.com/truenas/middleware/pull/9618
Bug Clerk August 11, 2022 at 2:59 PM
William Gryzbowski August 10, 2022 at 6:43 PM
Would be interesting to know what is the health of ctdb before rebooting the node: ctdb status
Complete
Pinned fields
Click on the next to a field label to start pinning.
Details
Details
Assignee
Triage TeamReporter
Joe MaloneyImpact
High
Fix versions
Affects versions
Priority
LowMore fields
Time tracking
More fields
Time trackingKatalon Platform
Linked Test Cases, Katalon Defect Results, Katalon Studio Test Results
Katalon Platform
Linked Test Cases, Katalon Defect Results, Katalon Studio Test Results
Created August 10, 2022 at 5:06 PM
Updated February 27, 2025 at 10:20 PM
Resolved August 11, 2022 at 6:00 PM
Created a fully working 4 node cluster with dispersed and a replicated volume. Shutdown 1/4 nodes to test failover. Failover worked. After powering back on the 4th node the 4th node could not rejoin the cluster.[2022/08/10 09:22:19] (ERROR) middlewared.job.run():423 - Job <bound method SharingSMBService.sync_registry of <middlewared.plugins.smb.SharingSMBService object at 0x7f0f68229f10>> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 411, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 446, in __run_body
rv = await self.method(([self] + args))
File "/usr/lib/python3/dist-packages/middlewared/plugins/smb.py", line 1761, in sync_registry
registry_shares = await self.middleware.call('sharing.smb.reg_listshares')
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1344, in call
return await self.call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1293, in call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/plugins/smb/registry_share.py", line 99, in reg_listshares
res = await self.netconf(action='list', jsoncmd=True)
File "/usr/lib/python3/dist-packages/middlewared/plugins/smb/registry_share.py", line 53, in netconf
raise CallError(
middlewared.service_exception.CallError: [ENXIO] Registry calls not permitted when ctdb unhealthy.
[2022/08/10 09:22:19] (DEBUG) EtcService.generate():468 - No new changes for /etc/default/rpcbind
[2022/08/10 09:22:22] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task alert.flush_alerts to run every 3600 seconds
[2022/08/10 09:22:22] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task alert.process_alerts to run every 60 seconds
[2022/08/10 09:22:22] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task certificate.renew_certs to run every 86400 seconds
[2022/08/10 09:22:22] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task certificateauthority.crl_generation to run every 86400 seconds
[2022/08/10 09:22:22] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task chart.release.periodic_chart_releases_update_checks to run every 86400 seconds
[2022/08/10 09:22:22] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task clusterjob.process_queue to run every 3600 seconds
[2022/08/10 09:22:22] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task kerberos.keytab.check_updated_keytab to run every 3600 seconds
[2022/08/10 09:22:22] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task kmip.sync_keys to run every 86400 seconds
[2022/08/10 09:22:22] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task mail.send_mail_queue to run every 600 seconds
[2022/08/10 09:22:22] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task pool.dataset.sync_db_keys to run every 86400 seconds
[2022/08/10 09:22:22] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task smb.sharesec.check_share_info_tdb to run every 3600 seconds
[2022/08/10 09:22:22] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task truecommand.health_check to run every 1800 seconds
[2022/08/10 09:22:22] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task zettarepl.flush_state to run every 3600 seconds
[2022/08/10 09:22:23] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:22:23] (ERROR) middlewared.job.run():423 - Job <bound method ClusterJob.process_queue of <middlewared.plugins.cluster_linux.ctdb_job.ClusterJob object at 0x7f0f681fd2e0>> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 411, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 446, in __run_body
rv = await self.method(([self] + args))
File "/usr/lib/python3/dist-packages/middlewared/plugins/cluster_linux/ctdb_job.py", line 63, in process_queue
node = (await self.middleware.call('ctdb.general.status', {'all_nodes': False}))['nodemap']['nodes'][0]
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1344, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1304, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1205, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1276, in nf
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1146, in nf
res = f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/cluster_linux/ctdb_general.py", line 138, in status
ctdb_status = ctdb.Client().status()
RuntimeError: /var/run/ctdb/ctdbd.socket: ctdb_client_init() failed: No such file or directory
[2022/08/10 09:22:23] (DEBUG) EtcService.generate():468 - No new changes for /etc/collectd/collectd.conf
[2022/08/10 09:22:24] (DEBUG) EtcService.render():47 - Failed to render mako template:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/etc.py", line 43, in render
return await self.service.middleware.run_in_thread(do)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1208, in run_in_thread
return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1205, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/etc.py", line 34, in do
return tmpl.render(
File "/usr/lib/python3/dist-packages/mako/template.py", line 476, in render
return runtime.render(self, self.callable, args, data)
File "/usr/lib/python3/dist-packages/mako/runtime.py", line 878, in _render
_render_context(
File "/usr/lib/python3/dist-packages/mako/runtime.py", line 920, in _render_context
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File "/usr/lib/python3/dist-packages/mako/runtime.py", line 947, in exec_template
callable(context, *args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/etc_files/local/avahi/avahi-daemon.conf.mako", line 21, in render_body
pnn = middleware.call_sync('ctdb.general.pnn')
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1374, in call_sync
return methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1276, in nf
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1146, in nf
res = f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/cluster_linux/ctdb_general.py", line 250, in pnn
self.this_node = ctdb.Client().pnn
RuntimeError: /var/run/ctdb/ctdbd.socket: ctdb_client_init() failed: No such file or directory
[2022/08/10 09:22:24] (ERROR) EtcService.generate():459 - Failed to render mako:local/avahi/avahi-daemon.conf
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/etc.py", line 449, in generate
rendered = await renderer.render(path, ctx)
File "/usr/lib/python3/dist-packages/middlewared/plugins/etc.py", line 43, in render
return await self.service.middleware.run_in_thread(do)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1208, in run_in_thread
return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1205, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/etc.py", line 34, in do
return tmpl.render(
File "/usr/lib/python3/dist-packages/mako/template.py", line 476, in render
return runtime.render(self, self.callable, args, data)
File "/usr/lib/python3/dist-packages/mako/runtime.py", line 878, in _render
_render_context(
File "/usr/lib/python3/dist-packages/mako/runtime.py", line 920, in _render_context
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File "/usr/lib/python3/dist-packages/mako/runtime.py", line 947, in exec_template
callable(context, *args, **kwargs)
File "/run/mako/etc_files/local/avahi/avahi-daemon.conf.mako.py", line 30, in render_body
pnn = middleware.call_sync('ctdb.general.pnn')
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1374, in call_sync
return methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1276, in nf
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1146, in nf
res = f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/cluster_linux/ctdb_general.py", line 250, in pnn
self.this_node = ctdb.Client().pnn
RuntimeError: /var/run/ctdb/ctdbd.socket: ctdb_client_init() failed: No such file or directory
[2022/08/10 09:22:25] (DEBUG) EtcService.generate():468 - No new changes for /etc/default/rrdcached
[2022/08/10 09:22:25] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:22:26] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:22:28] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:22:28] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:22:28] (DEBUG) EtcService.render():47 - Failed to render mako template:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/etc.py", line 43, in render
return await self.service.middleware.run_in_thread(do)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1208, in run_in_thread
return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1205, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/etc.py", line 34, in do
return tmpl.render(
File "/usr/lib/python3/dist-packages/mako/template.py", line 476, in render
return runtime.render(self, self.callable, args, data)
File "/usr/lib/python3/dist-packages/mako/runtime.py", line 878, in _render
_render_context(
File "/usr/lib/python3/dist-packages/mako/runtime.py", line 920, in _render_context
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File "/usr/lib/python3/dist-packages/mako/runtime.py", line 947, in exec_template
callable(context, *args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/etc_files/local/wsdd.conf.mako", line 8, in render_body
pnn = middleware.call_sync('ctdb.general.pnn')
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1374, in call_sync
return methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1276, in nf
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1146, in nf
res = f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/cluster_linux/ctdb_general.py", line 250, in pnn
self.this_node = ctdb.Client().pnn
RuntimeError: /var/run/ctdb/ctdbd.socket: ctdb_client_init() failed: No such file or directory
[2022/08/10 09:22:28] (ERROR) EtcService.generate():459 - Failed to render mako:local/wsdd.conf
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/etc.py", line 449, in generate
rendered = await renderer.render(path, ctx)
File "/usr/lib/python3/dist-packages/middlewared/plugins/etc.py", line 43, in render
return await self.service.middleware.run_in_thread(do)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1208, in run_in_thread
return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1205, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/etc.py", line 34, in do
return tmpl.render(
File "/usr/lib/python3/dist-packages/mako/template.py", line 476, in render
return runtime.render(self, self.callable, args, data)
File "/usr/lib/python3/dist-packages/mako/runtime.py", line 878, in _render
_render_context(
File "/usr/lib/python3/dist-packages/mako/runtime.py", line 920, in _render_context
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File "/usr/lib/python3/dist-packages/mako/runtime.py", line 947, in exec_template
callable(context, *args, **kwargs)
File "/run/mako/etc_files/local/wsdd.conf.mako.py", line 29, in render_body
pnn = middleware.call_sync('ctdb.general.pnn')
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1374, in call_sync
return methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1276, in nf
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1146, in nf
res = f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/cluster_linux/ctdb_general.py", line 250, in pnn
self.this_node = ctdb.Client().pnn
RuntimeError: /var/run/ctdb/ctdbd.socket: ctdb_client_init() failed: No such file or directory
[2022/08/10 09:22:30] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:22:30] (DEBUG) EtcService.generate():468 - No new changes for /etc/ctdb/ctdb.conf
[2022/08/10 09:22:30] (DEBUG) EtcService.generate():468 - No new changes for /etc/ctdb/ctdb.conf
[2022/08/10 09:22:43] (INFO) middlewared.init():838 - Starting TrueNAS-SCALE-22.02.3 middleware
[2022/08/10 09:22:46] (DEBUG) middlewared.setup():1483 - Timezone set to America/Los_Angeles
[2022/08/10 09:22:50] (DEBUG) middlewared.setup():25 - Certificate setup for System complete
[2022/08/10 09:22:50] (DEBUG) middlewared.__plugins_setup():955 - All plugins loaded
[2022/08/10 09:22:50] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task alert.flush_alerts to run every 3600 seconds
[2022/08/10 09:22:50] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task alert.process_alerts to run every 60 seconds
[2022/08/10 09:22:50] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task certificate.renew_certs to run every 86400 seconds
[2022/08/10 09:22:50] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task certificateauthority.crl_generation to run every 86400 seconds
[2022/08/10 09:22:50] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task chart.release.periodic_chart_releases_update_checks to run every 86400 seconds
[2022/08/10 09:22:50] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task clusterjob.process_queue to run every 3600 seconds
[2022/08/10 09:22:50] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task kerberos.keytab.check_updated_keytab to run every 3600 seconds
[2022/08/10 09:22:50] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task kmip.sync_keys to run every 86400 seconds
[2022/08/10 09:22:50] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task mail.send_mail_queue to run every 600 seconds
[2022/08/10 09:22:50] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task pool.dataset.sync_db_keys to run every 86400 seconds
[2022/08/10 09:22:50] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task smb.sharesec.check_share_info_tdb to run every 3600 seconds
[2022/08/10 09:22:50] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task truecommand.health_check to run every 1800 seconds
[2022/08/10 09:22:51] (DEBUG) middlewared._setup_periodic_tasks():968 - Setting up periodic task zettarepl.flush_state to run every 3600 seconds
[2022/08/10 09:22:51] (DEBUG) middlewared.__initialize():1758 - Accepting connections
[2022/08/10 09:22:51] (ERROR) DeviceService.get_rotational_rate():165 - Ioctl failed while retrieving rotational rate for disk /dev/vda
[2022/08/10 09:22:51] (ERROR) DeviceService.get_rotational_rate():165 - Ioctl failed while retrieving rotational rate for disk /dev/vdb
[2022/08/10 09:22:51] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:22:51] (ERROR) middlewared.job.run():423 - Job <bound method ClusterJob.process_queue of <middlewared.plugins.cluster_linux.ctdb_job.ClusterJob object at 0x7f81d9918580>> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 411, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 446, in __run_body
rv = await self.method(*([self] + args))
File "/usr/lib/python3/dist-packages/middlewared/plugins/cluster_linux/ctdb_job.py", line 63, in process_queue
node = (await self.middleware.call('ctdb.general.status', {'all_nodes': False}))['nodemap']['nodes'][0]
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1344, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1304, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1205, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1276, in nf
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1146, in nf
res = f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/cluster_linux/ctdb_general.py", line 138, in status
ctdb_status = ctdb.Client().status()
RuntimeError: /var/run/ctdb/ctdbd.socket: ctdb_client_init() failed: Connection refused
[2022/08/10 09:22:51] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:22:51] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:22:52] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:22:55] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:22:55] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:23:35] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:23:35] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:28:35] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:28:35] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:33:35] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:33:35] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:33:51] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:33:51] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:38:35] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:38:35] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:43:35] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:43:35] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:43:51] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:43:51] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy
[2022/08/10 09:47:18] (DEBUG) DirectoryServices.get_state():190 - Unable to determine directory services state while cluster is unhealthy