latency on the lo0
Description
Problem/Justification
Impact
SmartDraw Connector
Katalon Manual Tests (BETA)
Activity
Alexander Motin March 9, 2021 at 2:04 AMEdited
This is TrueNAS 12.0-U2.1 kernel with the fix for C-states latency, causing this ping latency issue and probably some others performance problems. Please update to 12.0-U2.1, create new boot environment, copy the archive somewhere on the system and replace the kernel with:
cd /boot
rm -r kernel
tar -xzvf /path/to/the/kernel_cx.tgz
reboot
Charles Orbello March 7, 2021 at 4:36 PM
Thanks Alexander for your help
Alexander Motin March 5, 2021 at 1:35 PM
It seems like an old FreeBSD problem in CPU idling, affecting idle systems with many cores, probably just triggered recently by changes in ICMP handling code, but not limited to it. I htink I see what is going on and have ideas how to fix. I've asked for comments some related FreeBSD developers.
Charles Orbello March 5, 2021 at 7:22 AM
Alexander, yes it's working
[~]# sysctl hw.acpi.cpu.cx_lowest=c1
hw.acpi.cpu.cx_lowest: C8 -> C1
[~]# ping -c 2 127.0.0.1
PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.096 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.097 ms
and if i i switch on C8 value
[~]# sysctl hw.acpi.cpu.cx_lowest=c8
hw.acpi.cpu.cx_lowest: C1 -> C8
[~]# ping -c 2 127.0.0.1
PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=8.673 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=12.227 ms
same problem with the value C2 and C3
Thanks
Alexander Motin March 4, 2021 at 10:37 PM
Charles, could you run `sysctl hw.acpi.cpu.cx_lowest=c1` and then test the ping latency? `sysctl hw.acpi.cpu.cx_lowest=c8` would revert where it is now.
Hi
i have a strange issue on my server Truenas 12 U1.2 , the ping latency on my local network is now very high (~10ms) but not only.
my server DL380 G10 is in prod since ~2years on Freenas without really pb
when i ping from a server the filer via the lan, i have a latency of ~10ms but if i use the arping it's ok.
by default the Lan is ok when i ping from the same server a another IP.
ping 192.168.200.20
PING 192.168.200.20 (192.168.200.20) 56(84) bytes of data.
64 bytes from 192.168.200.20: icmp_seq=1 ttl=64 time=0.386 ms
64 bytes from 192.168.200.20: icmp_seq=2 ttl=64 time=0.268 ms
64 bytes from 192.168.200.20: icmp_seq=3 ttl=64 time=0.229 ms
i restart my filer , my switch , same issue on a another network with a different switch.
if i start a arping from the same server to my filer it's ok
arping -I eno6 192.168.200.1
ARPING 192.168.200.1 from 192.168.200.10 eno6
Unicast reply from 192.168.200.1 [48:DF:37:AA:8B:93] 0.690ms
Unicast reply from 192.168.200.1 [48:DF:37:AA:8B:93] 0.722ms
Unicast reply from 192.168.200.1 [48:DF:37:AA:8B:93] 0.694ms
and now the more funny is if i ping my server via my filer
ping 192.168.200.10
PING 192.168.200.10 (192.168.200.10): 56 data bytes
64 bytes from 192.168.200.10: icmp_seq=0 ttl=64 time=2.597 ms
64 bytes from 192.168.200.10: icmp_seq=1 ttl=64 time=0.217 ms
64 bytes from 192.168.200.10: icmp_seq=2 ttl=64 time=0.265 ms
but if i ping my loopback on my filer i have the problem
ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=19.367 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=15.765 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=9.164 ms
same problem i create a new loopack interface
ifconfig lo1 create
ifconfig lo1 updated
ifconfig lo1 126.0.0.1
ping 126.0.0.1
PING 126.0.0.1 (126.0.0.1): 56 data bytes
64 bytes from 126.0.0.1: icmp_seq=0 ttl=64 time=11.958 ms
64 bytes from 126.0.0.1: icmp_seq=1 ttl=64 time=10.576 ms
the latency is ok if i boot on the iso install and start a shell.
and i reinstall the server from scratch the ping on the lo0 is of ~10ms
Rgds