mmap segfaulting on some files

Description

A reopen of Bug #47668 from the old system.

cp fails to copy some files with the error: Bad Address.

Our Case:

System uptime 84 days.

Copying approx 18000 files locally on the same ZFS share.

Randomly, approx 5 of them will fail with "Bad Adress". All are around 200kb in size.

cp: ./boost_1_73_0/boost/phoenix/object/detail/cpp03/preprocessed/construct_expr_50.hpp: Bad address
cp: ./boost_1_73_0/boost/phoenix/operator/detail/cpp03/preprocessed/mem_fun_ptr_eval_40.hpp: Bad address
cp: ./boost_1_73_0/boost/polygon/detail/polygon_arbitrary_formation.hpp: Bad address
cp: ./boost_1_73_0/boost/fusion/container/map/detail/cpp03/preprocessed/as_map50.hpp: Bad address

Truss output for one of the failed copies:

stat("<output file>",0x7fffffffe938) ERR#2 'No such file or directory'
openat(AT_FDCWD,"<input file>",O_RDONLY,00) = 3 (0x3)
openat(AT_FDCWD,"<output file>",O_WRONLY|O_CREAT|O_TRUNC,0100644) = 4 (0x4)
mmap(0x0,134681,PROT_READ,MAP_SHARED,3,0x0) = 34366312448 (0x800645000)
write(4,"/*=============================="...,134681) ERR#14 'Bad address'
write(2,"cp: ",4) = 4 (0x4)
write(2,"<output file trimmed>"...,83) = 83 (0x53)
write(2,": ",2) = 2 (0x2)
write(2,"Bad address\n",12) = 12 (0xc)
munmap(0x800645000,134681) = 0 (0x0)
futimens(4,{ 1589987039.485719267, 1589986667.732862967 }) = 0 (0x0)
fstat(4,{ mode=-rwxrwxrwx ,inode=2849020,size=131072,blksize=131072 }) = 0 (0x0)
fchown(4,138,11) = 0 (0x0)
fchmod(4,0644) = 0 (0x0)
fpathconf(0x3,0x40) = 1 (0x1)
__acl_get_fd(3,ACL_TYPE_NFS4,0x800ebc000) = 0 (0x0)
close(4) = 0 (0x0)
close(3) = 0 (0x0)

Problem/Justification

None

Impact

None

SmartDraw Connector

Katalon Manual Tests (BETA)

Activity

Show:

Alexander Motin October 14, 2020 at 6:48 PM

We haven't received any more reports like this.  Please retry it on upcoming TrueNAS 12.0, since it includes very different ZFS version and its new FreeBSD integration.

Chris Parsons May 26, 2020 at 12:28 AM

Debug file.

Chris Parsons May 26, 2020 at 12:17 AM

Not a files corrupted issue, zfs can read them, copying via dd or rsync works fine. Just processes that use mmap.

More info in the original ticket sitting in the old bug system, or this thread. https://www.ixsystems.com/community/threads/cp-bad-address-on-freenas-11-2-u6.82051/

will attach the debug.

Alexander Motin May 21, 2020 at 8:51 PM

Chris, can it be that those files are really somehow corrupted and ZFS does not allow to read them? Have you tried to scrub the pool, or try to read/copy them in some other way, not including mmap(), for example with `dd`?

You've mentioned 84 days uptime.  Does the problem persist if you reboot the system, i.e. is that a problem of specific file, or something wrong with already running system?

Please attach FreeNAS debug (System -> Advanced -> Save Debug).

Cannot Reproduce
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Labels

Components

Fix versions

Affects versions

Priority

More fields

Katalon Platform

Created May 21, 2020 at 12:38 AM
Updated July 1, 2022 at 4:53 PM
Resolved October 14, 2020 at 6:48 PM