[SCALE] SMB very slow and crashes on use

Description

Hey,

I just switch to SCALE Nightlies (TrueNAS-SCALE-20.11-MASTER-20201103-132915) from 12.1 Nightlies.

I noticed that under SCALE, SMB is very slow and somewhat unstable. If I copy larger files (or just overall) the connection will crash and I get these dumps in the CLI:

truenas# 2020 Nov 4 00:54:16 truenas Process 3203149 (smbd) of user 0 dumped co re.

Stack trace of thread 3203149:
#0 0x00007fc93e6b9db1 __GI_raise (libc.so.6 + 0x3bdb1)
#1 0x00007fc93e6a3537 __GI_abort (libc.so.6 + 0x25537)
#2 0x00007fc93eb8d610 dump_core (libsmbconf.so.0 + 0x3b610)
#3 0x00007fc93eb837c1 smb_panic_s3 (libsmbconf.so.0 + 0x317c1)
#4 0x00007fc93ec1a757 smb_panic (libsamba-util.so.0 + 0x21757)
#5 0x00007fc93ee84e02 n/a (libsmbd-base.so.0 + 0x208e02)
#6 0x00007fc93eab48e6 n/a (libtalloc.so.2 + 0x48e6)
#7 0x00007fc93edabdce n/a (libsmbd-base.so.0 + 0x12fdce)
#8 0x00007fc93edad5f2 n/a (libsmbd-base.so.0 + 0x1315f2)
#9 0x00007fc93edb0e3d create_file_default (libsmbd-base.so.0 + 0x134e3d)
#10 0x00007fc93edea515 smbd_smb2_request_process_create (libsmbd-base.so.0 + 0x1 6e515)
#11 0x00007fc93ede049d smbd_smb2_request_dispatch (libsmbd-base.so.0 + 0x16449d)
#12 0x00007fc93ede1dc7 n/a (libsmbd-base.so.0 + 0x165dc7)
#13 0x00007fc93e91f10d tevent_common_invoke_fd_handler (libtevent.so.0 + 0x710d)
#14 0x00007fc93e925497 n/a (libtevent.so.0 + 0xd497)
#15 0x00007fc93e923617 n/a (libtevent.so.0 + 0xb617)
#16 0x00007fc93e91e7b4 _tevent_loop_once (libtevent.so.0 + 0x67b4)
#17 0x00007fc93e91ea9b tevent_common_loop_wait (libtevent.so.0 + 0x6a9b)
#18 0x00007fc93e9235b7 n/a (libtevent.so.0 + 0xb5b7)
#19 0x00007fc93edceba0 smbd_process (libsmbd-base.so.0 + 0x152ba0)
#20 0x000055aef6763d85 n/a (smbd + 0xcd85)
#21 0x00007fc93e91f10d tevent_common_invoke_fd_handler (libtevent.so.0 + 0x710d)
#22 0x00007fc93e925497 n/a (libtevent.so.0 + 0xd497)
#23 0x00007fc93e923617 n/a (libtevent.so.0 + 0xb617)
#24 0x00007fc93e91e7b4 _tevent_loop_once (libtevent.so.0 + 0x67b4)
#25 0x00007fc93e91ea9b tevent_common_loop_wait (libtevent.so.0 + 0x6a9b)
#26 0x00007fc93e9235b7 n/a (libtevent.so.0 + 0xb5b7)
#27 0x000055aef675fb61 main (smbd + 0x8b61)
#28 0x00007fc93e6a4cca __libc_start_main (libc.so.6 + 0x26cca)
#29 0x000055aef675fdba _start (smbd + 0x8dba)

Problem/Justification

None

Impact

None

SmartDraw Connector

Katalon Manual Tests (BETA)

Activity

Show:

Andrew Walker January 1, 2021 at 1:02 AM

Okay. I will close this ticket as a duplicate of NAS-108814. At this point we are too early in the development cycle to have performance characterizations of what can be expected from your hardware. You can maybe test same hardware with TrueNAS 12.0-U1 and compare results.

Mike Tremaine December 31, 2020 at 4:01 PM
Edited

Unfortunately the error hasn't resurfaced on mine either. What does disabling shadow copies do?

Btw what were the original slowness issues you were seeing? I don't think I was seeing slowness normally... I have 4 drives w/a max of 260 MB/s in a RAIDZ configuration and I see ~380MB/s read and ~550 MB/s write, which seems expected. The interesting thing is I also have a single NVMe w/ZFS capable of  ~5000 MB/s reads and when I tried to copy over samba to another PC direct connected on a 10Gbe link to another NVMe (formatted as Ext4) I still only see ~380 MB/s. Is that expected? I would have expected NVMe->NVMe to max out that 10 Gbe link right?

Fraser December 31, 2020 at 1:09 PM

This seems to have fixed it for me. 200+GB transfer with no errors. Could not manage 20GB before. Will still monitor.

Thanks.

Andrew Walker December 31, 2020 at 1:02 AM

Hi guys, any further crashes to report?

Mike Tremaine December 29, 2020 at 4:57 PM

I also saw this crash in scale 20.12 and was about to file a ticket when I stumbled onto this, unfortunately I didn't have debug logs enabled and I only have the samba,samba4 and syslog from the crash. I have since enabled debug log messages and added the lock directory option so if it crashes again I'll update this ticket. My pools were all created in Scale 20.10, for what it's worth.

Duplicate
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Components

Fix versions

Affects versions

Priority

More fields

Katalon Platform

Created November 4, 2020 at 12:17 AM
Updated July 1, 2022 at 5:00 PM
Resolved January 1, 2021 at 1:03 AM