move_to_cgroups regression (zettarepl broken 100%)
Description
Problem/Justification
None
Impact
None
Activity
Show:
Muhammad Rehan May 27, 2022 at 2:16 PM
Closing this as we have not been able to reproduce and confirmed with Caleb as well to move in this direction for now until we have clarity on what might have happened on the specific system in question.
Cannot Reproduce
Pinned fields
Click on the next to a field label to start pinning.
Details
Details
Assignee
Triage Team
Triage TeamReporter
Caleb
CalebLabels
Time remaining
0m
Components
Fix versions
Affects versions
Priority
LowKatalon Platform
Linked Test Cases, Katalon Defect Results, Katalon Studio Test Results
Katalon Platform
Linked Test Cases, Katalon Defect Results, Katalon Studio Test Results
Created May 20, 2022 at 2:39 PM
Updated July 6, 2022 at 9:02 PM
Resolved May 27, 2022 at 2:16 PM
running version `22.12-MASTER-20220519-092905` "/var/log/middlewared.log" is being spammed with
2022/05/20 01:29:03] (ERROR) ZettareplService._join():327 - Abnormal zettarepl process termination with code 1, restarting
[2022/05/20 01:29:04] (ERROR) ZettareplService._join():327 - Abnormal zettarepl process termination with code 1, restarting
[2022/05/20 01:29:05] (ERROR) ZettareplService._join():327 - Abnormal zettarepl process termination with code 1, restarting
Further investigation shows it's failing here
May 20 07:37:18 r50b-1 middlewared[2135393]: Traceback (most recent call last):
May 20 07:37:18 r50b-1 middlewared[2135393]: File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
May 20 07:37:18 r50b-1 middlewared[2135393]: self.run()
May 20 07:37:18 r50b-1 middlewared[2135393]: File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
May 20 07:37:18 r50b-1 middlewared[2135393]: self._target(*self._args, **self._kwargs)
May 20 07:37:18 r50b-1 middlewared[2135393]: File "/usr/lib/python3/dist-packages/middlewared/plugins/zettarepl.py", line 147, in _call_
May 20 07:37:18 r50b-1 middlewared[2135393]: move_to_root_cgroups(os.getpid())
May 20 07:37:18 r50b-1 middlewared[2135393]: File "/usr/lib/python3/dist-packages/middlewared/utils/cgroups.py", line 8, in move_to_root_cgroups
May 20 07:37:18 r50b-1 middlewared[2135393]: with open(f"/sys/fs/cgroup/cgroup.procs", "w") as f2:
May 20 07:37:18 r50b-1 middlewared[2135393]: OSError: [Errno 30] Read-only file system: '/sys/fs/cgroup/cgroup.procs'
The most recent commit to this function was in 85f679538c9a48e0e29b5d3c1b666dd62e013642 which has broken zettarepl completely.
The only thing I've done on this box is edit an enclosure file (r50_nvme.py) and then "systemctl restart middlewared"