Replication with dedicated user broken TrueNAS Core -> TrueNAS Scale /usr/sbin is not in PATH
Description
Problem/Justification
Impact
relates to
Activity

Former user November 22, 2021 at 6:19 PMEdited
note:
nonprivileged user replication was setup with variation of this article
http://mikebellerue.blogspot.com/2016/06/freenas-replication-with-dedicated-user.html
the workaround to this issue was: on target system make sure replicator user shell is bash , then create .bashrc in replicator home directory with content:
export PATH=$PATH:/usr/sbin

Former user November 22, 2021 at 6:18 PM
Problem is visible on TrueNAS Core 12.0-U6/12.0-U6.1
With Push replication task run with nonprivileged user Pushing to TrueNAS-SCALE-22.02-RC.1-1
problem is related to fact that nonprivileged user doesn't have /sbin /usr/sbin in $PATH, and zfs command is in /usr/sbin/ in SCALE
and zetarepl is calling zfs without path
Attaching debug from booth systems,
example error from zetarepl
[2021/11/21 17:11:57] INFO [MainThread] [zettarepl.scheduler.clock] Interrupted
[2021/11/21 17:11:57] INFO [MainThread] [zettarepl.zettarepl] Scheduled tasks: [<Replication Task 'task_1'>]
[2021/11/21 17:11:58] INFO [Thread-9869] [zettarepl.paramiko.replication_task__task_1] Connected (version 2.0, client OpenSSH_8.4p1)
[2021/11/21 17:11:58] INFO [Thread-9869] [zettarepl.paramiko.replication_task__task_1] Authentication (publickey) successful!
[2021/11/21 17:11:58] ERROR [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' unhandled replication error ExecException(127, 'sh: 1: zfs: not found\n')
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/run.py", line 164, in run_replication_tasks
retry_stuck_replication(
File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/stuck.py", line 18, in retry_stuck_replication
return func()
File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/run.py", line 165, in <lambda>
lambda: run_replication_task_part(replication_task, source_dataset, src_context, dst_context,
File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/run.py", line 239, in run_replication_task_part
check_target_type(replication_task, source_dataset, src_context, dst_context)
File "/usr/local/lib/python3.9/site-packages/zettarepl/replication/run.py", line 269, in check_target_type
target_dataset_type = get_property(dst_context.shell, target_dataset, "type")
File "/usr/local/lib/python3.9/site-packages/zettarepl/transport/zfscli/_init_.py", line 108, in get_property
return get_properties(shell, dataset, {property: type}, include_source)[property]
File "/usr/local/lib/python3.9/site-packages/zettarepl/transport/zfscli/_init_.py", line 104, in get_properties
return get_properties_recursive(shell, [dataset], properties, include_source)[dataset]
File "/usr/local/lib/python3.9/site-packages/zettarepl/transport/zfscli/_init_.py", line 90, in get_properties_recursive
output = shell.exec(cmd)
File "/usr/local/lib/python3.9/site-packages/zettarepl/transport/interface.py", line 89, in exec
return self.exec_async(args, encoding, stdout).wait(timeout)
File "/usr/local/lib/python3.9/site-packages/zettarepl/transport/base_ssh.py", line 54, in wait
raise ExecException(exitcode, stdout)
zettarepl.transport.interface.ExecException: sh: 1: zfs: not found
[2021/11/21 17:12:00] INFO [retention] [zettarepl.zettarepl] Retention destroying local snapshots: []
[2021/11/21 17:12:00] INFO [Thread-9871] [zettarepl.paramiko.retention] Connected (version 2.0, client OpenSSH_8.4p1)
[2021/11/21 17:12:00] INFO [Thread-9871] [zettarepl.paramiko.retention] Authentication (publickey) successful!
[2021/11/21 17:12:00] WARNING [retention] [zettarepl.zettarepl] Remote retention failed on <SSH Transport(replicator@10.76.175.2)>: error listing snapshots: ExecException(127, 'sh: 1: zfs: not found\n')
[2021/11/21 17:17:29] INFO [MainThread] [zettarepl.scheduler.clock] Interrupted
[2021/11/21 17:17:29] INFO [MainThread] [zettarepl.zettarepl] Scheduled tasks: [<Replication Task 'task_1'>]
[2021/11/21 17:17:29] INFO [Thread-9873] [zettarepl.paramiko.replication_task__task_1] Connected (version 2.0, client OpenSSH_8.4p1)
[2021/11/21 17:17:30] INFO [Thread-9873] [zettarepl.paramiko.replication_task__task_1] Authentication (publickey) successful!
[2021/11/21 17:17:30] ERROR [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' unhandled replication error ExecException(127, 'sh: 1: zfs: not found\n')
Traceback (most recent call last):

Bonnie Follweiler November 22, 2021 at 6:01 PM
Thank you for the report, .
What version of TrueNAS are you seeing this issue in? Can you please attach a debug file to this ticket? To generate a debug file on TrueNAS CORE, log in to the TrueNAS web interface, go to System > Advanced, then click Save Debug and wait for the file to download to your local system. In TrueNAS SCALE, this option is in System Settings > Advanced.
Details
Assignee
Vladimir VinogradenkoVladimir VinogradenkoReporter
Former userFormer user(Deactivated)Labels
Impact
MediumTime remaining
0mFix versions
Priority
Low
Details
Details
Assignee

Reporter

I am utilising zetarepl using dedicated nonroot user.
i found tasks failing with error:
zfs command not found.
workaround was adding .bashrc for user
with:
export PATH=$PATH:/usr/sbin