Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Unexpected reboot/crash

Description

Hey,

so with the Nightlies 12.0-MASTER-202004100424 I got a Bug which caused to restart/crash my Server every day at around ~03:30AM.

The alerts only say that FreeNAS had an unexpected shutdown.

I'm not sure why, but maybe it has something to do with setting up the timezones?

[2020/04/14 03:46:10] (DEBUG) middlewared.setup():1527 - Timezone set to Europe/Berlin
[2020/04/14 03:46:13] (DEBUG) middlewared.setup():2802 - Certificate setup for System complete
[2020/04/14 03:46:13] (DEBUG) ZettareplService._observer_queue_reader():653 - Observer queue got DefinitionErrors([])
[2020/04/14 03:46:13] (DEBUG) middlewared.__plugins_setup():881 - All plugins loaded
[2020/04/14 03:46:13] (DEBUG) middlewared.__initialize():1449 - Accepting connections
[2020/04/14 03:46:16] (DEBUG) middlewared.sysctl_configuration():16 - Failed to set sysctl freenas.directoryservice.activedirectory.timeout.start -> 220: sysctl: unknown oid 'freenas.directoryservice.activedirectory.timeout.start'

[2020/04/14 03:46:16] (DEBUG) middlewared.sysctl_configuration():16 - Failed to set sysctl freenas.directoryservice.activedirectory.timeout.restart -> 400: sysctl: unknown oid 'freenas.directoryservice.activedirectory.timeout.restart'

[2020/04/14 03:46:17] (INFO) EtcService.generate_all():355 - Skipping nginx group generation
[2020/04/14 03:46:17] (ERROR) middlewared.setup():847 - System dataset is not mounted
[2020/04/14 03:46:17] (ERROR) middlewared.render_body():25 - Collectd configuration file could not be generated
[2020/04/14 03:46:17] (INFO) EtcService.generate_all():355 - Skipping system_dataset group generation
[2020/04/14 03:46:17] (INFO) EtcService.generate_all():355 - Skipping mdns group generation
[2020/04/14 03:46:18] (INFO) EtcService.generate_all():355 - Skipping syslogd group generation
[2020/04/14 03:46:18] (DEBUG) EtcService.generate():346 - No new changes for /etc/pam.d/sshd
[2020/04/14 03:46:18] (DEBUG) EtcService.generate():297 - mako:local/users.oath file removed.
[2020/04/14 03:46:18] (DEBUG) EtcService.generate():297 - mako:local/openvpn/server/openvpn_server.conf file removed.
[2020/04/14 03:46:18] (DEBUG) EtcService.generate():297 - mako:local/openvpn/client/openvpn_client.conf file removed.
[2020/04/14 03:46:18] (DEBUG) middlewared.interface_pre_sync_hook():1531 - No HA hardware detected, skipping interfaces setup.
[2020/04/14 03:46:18] (INFO) InterfaceService.sync():1731 - Interfaces in database: ix0
[2020/04/14 03:46:18] (DEBUG) InterfaceService.configure():142 - ix0: adding 192.168.188.29/255.255.255.0
[2020/04/14 03:46:19] (DEBUG) InterfaceService.configure():175 - Starting dhclient for ix0
[2020/04/14 03:46:26] (DEBUG) EtcService.generate():346 - No new changes for /etc/dhclient.conf
[2020/04/14 03:46:28] (DEBUG) EtcService.generate():346 - No new changes for /etc/nsswitch.conf
[2020/04/14 03:46:28] (DEBUG) EtcService.generate():346 - No new changes for /etc/local/nslcd.conf
[2020/04/14 03:46:41] (DEBUG) middlewared.dumpdev_configure():24 - Configured "/dev/ada3p1" device for crash dumps.
[2020/04/14 03:46:44] (DEBUG) ZettareplService._observer_queue_reader():653 - Observer queue got DefinitionErrors([])
[2020/04/14 03:46:45] (DEBUG) EtcService.generate():346 - No new changes for /etc/local/smb4_share.conf
[2020/04/14 03:46:45] (DEBUG) EtcService.generate():346 - No new changes for /etc/krb5.conf
[2020/04/14 03:46:45] (DEBUG) EtcService.generate():346 - No new changes for /etc/krb5.conf
[2020/04/14 03:46:45] (DEBUG) middlewared._setup_periodic_tasks():894 - Setting up periodic task alert.flush_alerts to run every 3600 seconds
[2020/04/14 03:46:45] (DEBUG) middlewared._setup_periodic_tasks():894 - Setting up periodic task alert.process_alerts to run every 60 seconds
[2020/04/14 03:46:45] (DEBUG) middlewared._setup_periodic_tasks():894 - Setting up periodic task certificate.renew_certs to run every 86400 seconds
[2020/04/14 03:46:45] (DEBUG) middlewared._setup_periodic_tasks():894 - Setting up periodic task certificateauthority.crl_generation to run every 86400 seconds
[2020/04/14 03:46:45] (DEBUG) middlewared._setup_periodic_tasks():894 - Setting up periodic task kerberos.keytab.check_updated_keytab to run every 3600 seconds
[2020/04/14 03:46:45] (DEBUG) middlewared._setup_periodic_tasks():894 - Setting up periodic task kmip.sync_keys to run every 86400 seconds
[2020/04/14 03:46:45] (DEBUG) middlewared._setup_periodic_tasks():894 - Setting up periodic task mail.send_mail_queue to run every 600 seconds
[2020/04/14 03:46:45] (DEBUG) middlewared._setup_periodic_tasks():894 - Setting up periodic task plugin.periodic_plugin_update to run every 86400 seconds
[2020/04/14 03:46:45] (DEBUG) middlewared._setup_periodic_tasks():894 - Setting up periodic task plugin.retrieve_versions_for_repos to run every 86400 seconds
[2020/04/14 03:46:45] (DEBUG) middlewared._setup_periodic_tasks():894 - Setting up periodic task pool.dataset.sync_db_keys to run every 86400 seconds
[2020/04/14 03:46:45] (DEBUG) middlewared._setup_periodic_tasks():894 - Setting up periodic task smb.sharesec.check_share_info_tdb to run every 3600 seconds
[2020/04/14 03:46:45] (DEBUG) middlewared._setup_periodic_tasks():894 - Setting up periodic task zettarepl.flush_state to run every 3600 seconds
[2020/04/14 03:46:46] (DEBUG) JailService.start_on_boot():1625 - Starting jails on boot: PENDING
[2020/04/14 03:46:46] (DEBUG) JailService.start_on_boot():1627 - Starting jails on boot: SUCCESS
[2020/04/14 03:46:47] (DEBUG) EtcService.generate():346 - No new changes for /etc/local/smb4.conf
[2020/04/14 03:46:47] (DEBUG) EtcService.generate():346 - No new changes for /etc/security/pam_winbind.conf
[2020/04/14 03:46:47] (DEBUG) urllib3.connectionpool._new_conn():225 - Starting new HTTP connection (1): 192.168.188.29:443
[2020/04/14 03:46:47] (DEBUG) urllib3.connectionpool._make_request():437 - http://192.168.188.29:443 "HEAD / HTTP/1.1" 400 0
[2020/04/14 03:46:47] (DEBUG) urllib3.connectionpool._new_conn():959 - Starting new HTTPS connection (1): 192.168.188.29:443
/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning,
[2020/04/14 03:46:47] (DEBUG) urllib3.connectionpool._make_request():437 - https://192.168.188.29:443 "HEAD / HTTP/1.1" 302 0
[2020/04/14 03:46:47] (DEBUG) urllib3.connectionpool._new_conn():225 - Starting new HTTP connection (1): 192.168.188.29:80
[2020/04/14 03:46:47] (DEBUG) urllib3.connectionpool._make_request():437 - http://192.168.188.29:80 "HEAD / HTTP/1.1" 307 0
[2020/04/14 03:46:47] (DEBUG) urllib3.connectionpool._new_conn():959 - Starting new HTTPS connection (1): 192.168.188.29:80
[2020/04/14 03:46:47] (DEBUG) EtcService.generate():346 - No new changes for /etc/local/smb4_share.conf
[2020/04/14 03:46:48] (DEBUG) middlewared.status():1498 - Failed to retrieve VM status for Torrent
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/vm.py", line 1496, in status
return self.vms[vm['name']].status()
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/vm.py", line 151, in _getattribute_
raise RuntimeError('Domain attribute not defined, please re-instantiate the VM class')
RuntimeError: Domain attribute not defined, please re-instantiate the VM class
[2020/04/14 03:46:49] (DEBUG) EtcService.generate():346 - No new changes for /etc/local/avahi/avahi-daemon.conf
[2020/04/14 03:46:50] (DEBUG) EtcService.generate():346 - No new changes for /etc/local/avahi/avahi-daemon.conf
[2020/04/14 03:46:50] (DEBUG) SMBService.add_admin_group():86 - No cache entry indicating delayed action to add admin_group was found.
[2020/04/14 03:47:47] (ERROR) iocage.callback():66 - Failed to update git repository: fatal: unable to access 'https://github.com/ix-plugin-hub/iocage-plugin-index.git/': Could not resolve host: github.com

[2020/04/14 03:47:47] (ERROR) iocage.callback():66 - Not cloning https://github.com/ix-plugin-hub/iocage-plugin-index.gitas git-pull failed due to network issues.
[2020/04/14 03:47:47] (DEBUG) urllib3.connectionpool._new_conn():225 - Starting new HTTP connection (1): pkg.freebsd.org:80
[2020/04/14 03:47:47] (DEBUG) urllib3.connectionpool._new_conn():225 - Starting new HTTP connection (1): pkg.freebsd.org:80
[2020/04/14 03:47:47] (DEBUG) urllib3.connectionpool._new_conn():225 - Starting new HTTP connection (1): pkg.cdn.trueos.org:80
[2020/04/14 03:47:47] (DEBUG) urllib3.connectionpool._new_conn():225 - Starting new HTTP connection (1): pkg.cdn.trueos.org:80
[2020/04/14 03:48:47] (ERROR) iocage.callback():66 - Failed to update git repository: fatal: unable to access 'https://github.com/freenas/iocage-ix-plugins.git/': Could not resolve host: github.com

[2020/04/14 03:48:47] (ERROR) iocage.callback():66 - Not cloning https://github.com/freenas/iocage-ix-plugins.gitas git-pull failed due to network issues.
[2020/04/14 03:48:47] (DEBUG) urllib3.connectionpool._new_conn():225 - Starting new HTTP connection (1): pkg.freebsd.org:80
[2020/04/14 03:48:47] (DEBUG) urllib3.connectionpool._new_conn():225 - Starting new HTTP connection (1): pkg.cdn.trueos.org:80
[2020/04/14 03:49:49] (ERROR) iocage.callback():66 - Failed to update git repository: fatal: unable to access 'https://github.com/ix-plugin-hub/iocage-plugin-index.git/': Could not resolve host: github.com

[2020/04/14 03:49:49] (ERROR) iocage.callback():66 - Not cloning https://github.com/ix-plugin-hub/iocage-plugin-index.gitas git-pull failed due to network issues.
[2020/04/14 03:50:50] (ERROR) iocage.callback():66 - Failed to update git repository: fatal: unable to access 'https://github.com/freenas/iocage-ix-plugins.git/': Could not resolve host: github.com

[2020/04/14 03:50:50] (ERROR) iocage.callback():66 - Not cloning https://github.com/freenas/iocage-ix-plugins.gitas git-pull failed due to network issues.

Problem/Justification

None

Impact

None

SmartDraw Connector

Katalon Manual Tests (BETA)

Activity

Show:

Ryan Moeller April 17, 2020 at 11:43 PM

Fix merged, should be in an upcoming nightly. To reiterate, you probably shouldn't be using the debug kernel in the first place, but that panic should not happen again. Let us know if you still encounter a problem.

Ryan Moeller April 16, 2020 at 3:59 AM

Ryan Moeller April 16, 2020 at 3:57 AM

This should be fixed soon, but FWIW the problem should go away if you use the regular kernel instead of the debug kernel (system/advanced/kernel)

Anda April 16, 2020 at 2:00 AM

Just got It again with version "TrueNAS-12.0-MASTER-202004090424".

And the second one:

I now rollback to 12.0-MASTER-202003250424 as this was the last stable version for me. New debug-> 

Complete
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Components

Fix versions

Affects versions

Priority

More fields

Katalon Platform

Created April 14, 2020 at 11:57 AM
Updated July 1, 2022 at 4:49 PM
Resolved April 17, 2020 at 11:43 PM
Configure