Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Freenas hangs after restoring timemachine permissions and starting timemachine backup

Description

My timemachine backups stopped working after the upgrade to 11.1 or 11.2
I have tried to revive them which is probably related to the permission of the AFP share (all permissions were gone, might be related to a migration issue).
When I restored the permission and started timemachine, freenas hangs and I could only get access to it by resetting it through ipmi (UI and ssh were not responsive any more).

In the ipmi console I see the error message: swap_pager_getswapspace(xx): failed
In middlewared.log I see the following error around the time of the issue.

[2019/12/10 08:59:29] (DEBUG) ServiceService._simplecmd():281 - Calling: reload(afp)
[2019/12/10 09:00:41] (WARNING) middlewared._loop_monitor_thread():1203 - Task seems blocked:
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1271, in run
self.__loop.run_forever()
File "/usr/local/lib/python3.6/asyncio/base_events.py", line 438, in run_forever
self._run_once()
File "/usr/local/lib/python3.6/asyncio/base_events.py", line 1451, in _run_once
handle._run()
File "/usr/local/lib/python3.6/asyncio/events.py", line 145, in _run
self._callback(*self._args)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 165, in call_method
result = await self.middleware.call_method(self, message)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1096, in call_method
return await self._call(message['method'], serviceobj, methodobj, params, app=app, io_thread=False)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1044, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 664, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/service.py", line 201, in config
return await self._get_or_insert(self._config.datastore, options)
File "/usr/local/lib/python3.6/site-packages/middlewared/service.py", line 211, in _get_or_insert
return await self.middleware.call('datastore.config', datastore, options)

[2019/12/10 09:00:43] (WARNING) middlewared._loop_monitor_thread():1203 - Task seems blocked:
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1271, in run
self.__loop.run_forever()
File "/usr/local/lib/python3.6/asyncio/base_events.py", line 438, in run_forever
self._run_once()
File "/usr/local/lib/python3.6/asyncio/base_events.py", line 1451, in _run_once
handle._run()
File "/usr/local/lib/python3.6/asyncio/events.py", line 145, in _run
self._callback(*self._args)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 165, in call_method
result = await self.middleware.call_method(self, message)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1096, in call_method
return await self._call(message['method'], serviceobj, methodobj, params, app=app, io_thread=False)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1044, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 664, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/service.py", line 201, in config
return await self._get_or_insert(self._config.datastore, options)
File "/usr/local/lib/python3.6/site-packages/middlewared/service.py", line 211, in _get_or_insert
return await self.middleware.call('datastore.config', datastore, options)

[2019/12/10 09:01:03] (WARNING) middlewared._loop_monitor_thread():1200 - run_in_thread seems to be blocking event loop
[2019/12/10 09:01:05] (WARNING) middlewared._loop_monitor_thread():1200 - run_in_thread seems to be blocking event loop
[2019/12/10 09:01:07] (WARNING) middlewared._loop_monitor_thread():1200 - run_in_thread seems to be blocking event loop

Problem/Justification

None

Impact

None

SmartDraw Connector

Katalon Manual Tests (BETA)

Activity

Show:

Andrew Walker December 16, 2019 at 3:29 PM

>Is it possible that a similar prevents other services from working well? And how do I check that?

If there is resource contention between the docker containers / VMs and services such as time machine then, yes, it can impact service stability. There are many ways of monitoring resource utilization. Feel free to check out our documentation and community forums if you need help with this.

Ton S December 16, 2019 at 2:36 PM
Edited

Timemachine is running again! The freenas UI is still responsive. I only stopped my docker containers (and my virtual machines). I doubt if it is required for timemachine to run.

Thanks a lot. I could not have this myself.

Is it possible that a similar prevents other services from working well? And how do I check that?

Andrew Walker December 16, 2019 at 11:12 AM

Now chmod 755 /mnt/Tank and restart the time machine backup.

Ton S December 13, 2019 at 9:36 PM

When I run setfacl it runs fine for a few times and every time a line disappears.
Except for the last item, I get an error message with setfacl.

root@freenas:/mnt # setfacl -x 0 /mnt/Tank
setfacl: /mnt/Tank: acl_set_file() failed: Invalid argument

root@freenas:/mnt # getfacl /mnt/Tank

  1. file: /mnt/Tank

  2. owner: root

  3. group: wheel
    everyone@:r-x---a-R-c--s:-------:allow

 

 

Andrew Walker December 13, 2019 at 12:23 PM

Your permissions on /mnt/Tank appear to be broken

Let's try very thoroughly nuking and resetting them. Type the following in the terminal
setfacl -x 0 /mnt/Tank
press enter, then press the up arrow again to repeat the same command. Do this until getfacl /mnt/Tank shows nothing. Once you have done this run the following command:
chmod 755 /mnt/Tank

Other than that I don't see anything immediately actionable. The logs show the FreeNAS getting swapped to death. I see that you're on version 11.2-U6. Once you've fixed the permissions on /mnt/Tank, re-try time machine, but this time stop your jails and bhyve VMs for the duration of the Time Machine run.

User Configuration Error
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Components

Fix versions

Affects versions

Priority

More fields

Katalon Platform

Created December 10, 2019 at 9:54 AM
Updated July 1, 2022 at 3:51 PM
Resolved December 16, 2019 at 3:29 PM
Configure