Forums

tomas
tomas
Offline
Resolved
0 votes
Hi there,

Setting up a new server and have an issue with one NIC -> Intel PRO/1000 GT - found out after I was doing some testing with iperf.

I'm getting low speed compared to other NICs (just 560 Mbits/sec, 960 Mbits/sec other two Realtek based) and plenty of entries in messages.log like:

AMD-Vi: Event logged [IO_PAGE_FAULT device=05:00.0 domain=0x000e address=0x0000000007c4c100 flags=0x0000]


lspci



00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1450
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Device 1451
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1460
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1461
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1462
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1463
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1464
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1465
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1466
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1467
01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] USB 3.1 XHCI Controller (rev 02)
01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b7 (rev 02)
01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b2 (rev 02)
02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
02:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
02:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
02:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
04:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 04)
05:01.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)
08:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 01)
09:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 710B] (rev a1)
09:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a
0a:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1456
0a:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] USB3 Host Controller
0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455
0b:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
0b:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1457


The other hardware:

AMD Ryzen 5 1600
Corsair CMK8GX4M2B3000C15 Vengeance LPX 8 GB (2 x 4 GB) at 2993MHz
Asus Prime B350-PLUS
GPU - Palit NVIDIA GeForce GT 710 (had to put something :-) )
Samsung 850 EVO 500 GB 2x (RAID 1)

I have tried 2 Intel PRO/1000 GT NICs as I had a spare. The install is fresh. First thing I decided to do is test NICs.

Is is a driver or something else? These NICs used to work perfectly with ClearOS 6.x but with different hardware ie Intel CPU, motherboard etc.


Help would be appreciated.

Edit - corrected typo/changed subject
Thursday, May 10 2018, 03:49 PM
Share this post:
Responses (12)
  • Accepted Answer

    Saturday, May 12 2018, 08:02 AM - #Permalink
    Resolved
    0 votes
    It is not quite as bad as that. Kernel updates only take effect when you reboot and not when they install. Also, if you don't recompile, it will just revert to the standard driver so you'd be no worse off than you are now. ...... but that is even if you can get the driver to compile as it is doing something wrong with its version checking in the Makefile.
    The reply is currently minimized Show
  • Accepted Answer

    tomas
    tomas
    Offline
    Saturday, May 12 2018, 07:44 AM - #Permalink
    Resolved
    0 votes
    Thanks, Nick.

    The problem is that I don't think it is kmod compatible so for every kernel update you will need to recompile it.


    Hm...Not perfect for production with automatic updates enabled. Will consider giving it a go just to find out if it sorts the issue. In the long term I might just leave it as it is and use the e1000 NIC as a backup due to poor performance. If I'm not mistaken IOMMU can be disabled as it's used for virtualization which I don't need. Another option would be to get another Realtek NIC and enable IOMMU.

    BTW for all Ryzen users - update your BIOS (release with AGESA 1002a). Then enable "C-power states" if you disabled it to sort out crashes/hang out and set under "Advanced/AMD CBS" -> "Power Supply Idle Control" to "Typical current idle" - 18h up and running with this. Previously every idle (30 min or so) I had a crash. Looks like "Typical current idle" setting sorts it out and with "C-power states" enabled you have some C states / power saving features.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, May 11 2018, 07:52 PM - #Permalink
    Resolved
    0 votes
    ......... And if you get past that part of the makefile there is a similar test further down.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, May 11 2018, 04:48 PM - #Permalink
    Resolved
    0 votes
    There is a newer driver on Intel's site which should work - it is the same version as the latest kmod one for ClearOS 6.x. It is years since I have compiled directly from Intel souces, but, from memory they compiled nicely, either following their README or by building an rpm with "rpm -tb ....." then installing the RPM.

    The problem is that I don't think it is kmod compatible so for every kernel update you will need to recompile it.

    Also I've just tried compiling it and it fails with a false error from the Makefile saying:
    Makefile:198: *** *** Aborting the build. *** This driver is not supported on kernel versions older than 2.4.0.  Stop.
    .... but the kernel version is 3.10.693 so it should pass. Perhaps that part of the makefile needs bypassing but you can try.
    The reply is currently minimized Show
  • Accepted Answer

    tomas
    tomas
    Offline
    Friday, May 11 2018, 12:00 PM - #Permalink
    Resolved
    0 votes
    Tried that and adding "amd_iommu=pt" to grub doesn't help.

    What helps though is disabling IOMMU in BIOS - no more "AMD-Vi: Event logged [IO_PAGE_FAULT..." messages.

    The performance of Intel PRO/1000 GT is disappointing though as I'm getting around 701 Mbits/sec speeds (500 Mbits/sec with IOMMU enabled) and over 940 Mbits/sec using Realtek based NICs. On 6.x Intel build it was the best NIC. I think this is actually kernel and/or E1000 driver issue as I have read posts that this type of issues stop occurring on latest 4.x Linux kernels.

    With disabled IOMMU in BIOS Realtek based NICs don't seem to lose performance. Not sure ATM if any other hardware is affected by IOMMU being disabled.

    Is there a newer e1000 driver I could try?
    The reply is currently minimized Show
  • Accepted Answer

    Friday, May 11 2018, 11:38 AM - #Permalink
    Resolved
    0 votes
    The safest thing to do is try it first. When you boot ClearOS it generally shows you a page with a number of kernels to boot from and has a 5s countdown. Interrupt the countdown by hitting any key then follow the on-screen instructions. I think you position your cursor on the kernel you want (the latest and is normally already selected) then hit 'e'. Scroll down to the long "linuxefi" or "linux" line (depending if you are doing an EFI boot), add your parameter to the end then Ctl-X to start ClearOS. This only affects the current boot and goes back to default on the following reboot.

    Only if that works is it worth changing grub2.
    The reply is currently minimized Show
  • Accepted Answer

    tomas
    tomas
    Offline
    Friday, May 11 2018, 10:54 AM - #Permalink
    Resolved
    0 votes
    Hm...Based based on Google search for "AMD-Vi: Event logged [IO_PAGE_FAULT e1000" issue is not with a driver.

    e.g

    http://iommu.linux-foundation.narkive.com/0F3L0C3K/e1000-devel-amd-vi-event-logged-io-page-fault-ixgbe-detected-tx-unit-hang-reset-adapter-master

    and to sort it out "/etc/default/grub" file would need to be modified and the following added:

    amd_iommu=pt


    https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.1/html/installation_guide/appe-configuring_a_hypervisor_host_for_pci_passthrough

    I have never modified this file. ATM it contains:

    GRUB_TIMEOUT=5
    GRUB_DISTRIBUTOR="ClearOS"
    GRUB_DEFAULT=saved
    GRUB_DISABLE_SUBMENU=true
    GRUB_TERMINAL_OUTPUT="console"
    GRUB_CMDLINE_LINUX="rd.md.uuid=5f26894e:78def273:30acb8f2:4c20a4ef rd.md.uuid=7cfd15e3:677c1a29:9a1acfca:cab7bde2 rhgb quiet"
    GRUB_DISABLE_RECOVERY="true"


    Is this automatically generated and will be replaced after e.g. kernel update? Or it's safe to edit and add
    amd_iommu=pt
    ?

    Thanks.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, May 11 2018, 10:15 AM - #Permalink
    Resolved
    0 votes
    The way of blacklisting modules changed in v7. Installing the kmod-r8169 removes the r8168 from the r8169's driver compatibility list which removes the need for blacklisting.

    Re the e1000, I have no idea and all I can do is google it which it looks like you've been doing anyway.
    The reply is currently minimized Show
  • Accepted Answer

    tomas
    tomas
    Offline
    Friday, May 11 2018, 09:08 AM - #Permalink
    Resolved
    0 votes
    After rebooting the driver r8168 is not loaded. So going with previous methods I have added

    "blacklist r8169"

    to /etc/modprobe.d/clearos.conf and created NIC.conf in the same folder (644) containing:

    alias enp3s0 r8168
    alias enp8s0 r8168

    but this doesn't seem to work too. Now I'm having even more issues :-(


    This was not needed. I have also installed kmod kmod-r8169 which sorted the issues with Realtek.

    Output of lspci -knn | grep Eth -A 3 now:

    03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
    Subsystem: ASUSTeK Computer Inc. Device [1043:8677]
    Kernel driver in use: r8168
    Kernel modules: r8168
    --
    05:01.0 Ethernet controller [0200]: Intel Corporation 82541PI Gigabit Ethernet Controller [8086:107c] (rev 05)
    Subsystem: Intel Corporation PRO/1000 GT Desktop Adapter [8086:1376]
    Kernel driver in use: e1000
    Kernel modules: e1000
    08:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 01)
    Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller [10ec:8168]
    Kernel driver in use: r8168
    Kernel modules: r8168
    09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 710B] [10de:128b] (rev a1)


    The issue with e1000 still remains.
    The reply is currently minimized Show
  • Accepted Answer

    tomas
    tomas
    Offline
    Friday, May 11 2018, 08:43 AM - #Permalink
    Resolved
    0 votes
    I do have one question, though. Are you sure the Ryzen5 is compatible with EL7? I am not sure. EL7 is often not too up to date with hardware support.


    I did some research before purchasing - basically I wanted new-gen hardware as I plan to keep this server for years. It was between Ryzen (1st gen) or Intel i5 8th gen. As I wasn't sure about compatibility of Intel 8th gen CPUs and mobos with EL7 I went with Ryzen (1st gen). There was a big increase in performance in 2017 thanks to AMD releasing Ryzen CPUs and Intel responding with 8th gen CPUs. Yes, there was a risk involved with Ryzen - I know about the crashes related to C6 states and I had few already so I disabled "C-power states" in BIOS to see if it helps. I will also run segfault test. Honestly I was expecting things to be sorted out as it's been more than a year since Ryzen 1st gen was released.

    Unless you've done something about it, you probably also have issues with the RTL8111/8168/8411 and strongly recommend you install the kmod-r8168 and kmod-r8169 drivers then reboot.


    Not yet. Last time I had to deal with it was with ClearOS 5.x and 6.x.

    Output of lspci -knn | grep Eth -A 3

    [root@system2 ~]# lspci -knn | grep Eth -A 3
    03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
    Subsystem: ASUSTeK Computer Inc. Device [1043:8677]
    Kernel driver in use: r8169
    Kernel modules: r8169
    --
    05:01.0 Ethernet controller [0200]: Intel Corporation 82541PI Gigabit Ethernet Controller [8086:107c] (rev 05)
    Subsystem: Intel Corporation PRO/1000 GT Desktop Adapter [8086:1376]
    Kernel driver in use: e1000
    Kernel modules: e1000
    08:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 01)
    Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller [10ec:8168]
    Kernel driver in use: r8169
    Kernel modules: r8169
    09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 710B] [10de:128b] (rev a1)


    Looking at this both Realtek based NICs are "8168".


    Output of modinfo e1000

    [root@system2 ~]# modinfo e1000
    filename: /lib/modules/3.10.0-693.2.2.v7.x86_64/kernel/drivers/net/ethernet/intel/e1000/e1000.ko.xz
    version: 7.3.21-k8-NAPI
    license: GPL
    description: Intel(R) PRO/1000 Network Driver
    author: Intel Corporation, <linux.nics@intel.com>
    rhelversion: 7.4
    srcversion: 9E0A112E5D47C996E7C4A58
    alias: pci:v00008086d00002E6Esv*sd*bc*sc*i*
    alias: pci:v00008086d000010B5sv*sd*bc*sc*i*
    alias: pci:v00008086d00001099sv*sd*bc*sc*i*
    alias: pci:v00008086d0000108Asv*sd*bc*sc*i*
    alias: pci:v00008086d0000107Csv*sd*bc*sc*i*
    alias: pci:v00008086d0000107Bsv*sd*bc*sc*i*
    alias: pci:v00008086d0000107Asv*sd*bc*sc*i*
    alias: pci:v00008086d00001079sv*sd*bc*sc*i*
    alias: pci:v00008086d00001078sv*sd*bc*sc*i*
    alias: pci:v00008086d00001077sv*sd*bc*sc*i*
    alias: pci:v00008086d00001076sv*sd*bc*sc*i*
    alias: pci:v00008086d00001075sv*sd*bc*sc*i*
    alias: pci:v00008086d00001028sv*sd*bc*sc*i*
    alias: pci:v00008086d00001027sv*sd*bc*sc*i*
    alias: pci:v00008086d00001026sv*sd*bc*sc*i*
    alias: pci:v00008086d0000101Esv*sd*bc*sc*i*
    alias: pci:v00008086d0000101Dsv*sd*bc*sc*i*
    alias: pci:v00008086d0000101Asv*sd*bc*sc*i*
    alias: pci:v00008086d00001019sv*sd*bc*sc*i*
    alias: pci:v00008086d00001018sv*sd*bc*sc*i*
    alias: pci:v00008086d00001017sv*sd*bc*sc*i*
    alias: pci:v00008086d00001016sv*sd*bc*sc*i*
    alias: pci:v00008086d00001015sv*sd*bc*sc*i*
    alias: pci:v00008086d00001014sv*sd*bc*sc*i*
    alias: pci:v00008086d00001013sv*sd*bc*sc*i*
    alias: pci:v00008086d00001012sv*sd*bc*sc*i*
    alias: pci:v00008086d00001011sv*sd*bc*sc*i*
    alias: pci:v00008086d00001010sv*sd*bc*sc*i*
    alias: pci:v00008086d0000100Fsv*sd*bc*sc*i*
    alias: pci:v00008086d0000100Esv*sd*bc*sc*i*
    alias: pci:v00008086d0000100Dsv*sd*bc*sc*i*
    alias: pci:v00008086d0000100Csv*sd*bc*sc*i*
    alias: pci:v00008086d00001009sv*sd*bc*sc*i*
    alias: pci:v00008086d00001008sv*sd*bc*sc*i*
    alias: pci:v00008086d00001004sv*sd*bc*sc*i*
    alias: pci:v00008086d00001001sv*sd*bc*sc*i*
    alias: pci:v00008086d00001000sv*sd*bc*sc*i*
    depends:
    intree: Y
    vermagic: 3.10.0-693.2.2.v7.x86_64 SMP mod_unload modversions
    signer: CentOS Linux kernel signing key
    sig_key: F2:AA:4B:11:57:E9:44:B9:80:7F:78:DC:00:11:6E:E1:EC:9A:04:0C
    sig_hashalgo: sha256
    parm: TxDescriptors:Number of transmit descriptors (array of int)
    parm: RxDescriptors:Number of receive descriptors (array of int)
    parm: Speed:Speed setting (array of int)
    parm: Duplex:Duplex setting (array of int)
    parm: AutoNeg:Advertised auto-negotiation setting (array of int)
    parm: FlowControl:Flow Control setting (array of int)
    parm: XsumRX:Disable or enable Receive Checksum offload (array of int)
    parm: TxIntDelay:Transmit Interrupt Delay (array of int)
    parm: TxAbsIntDelay:Transmit Absolute Interrupt Delay (array of int)
    parm: RxIntDelay:Receive Interrupt Delay (array of int)
    parm: RxAbsIntDelay:Receive Absolute Interrupt Delay (array of int)
    parm: InterruptThrottleRate:Interrupt Throttling Rate (array of int)
    parm: SmartPowerDownEnable:Enable PHY smart power down (array of int)
    parm: copybreak:Maximum size of packet that is copied to a new buffer on receive (uint)
    parm: debug:Debug level (0=none,...,16=all) (int)


    I have installed vanilla kmod-r8168 (yum install kmod-r8168)

    lspci -knn | grep Eth -A 3 now:

    [root@system2 ~]# lspci -knn | grep Eth -A 3
    03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
    Subsystem: ASUSTeK Computer Inc. Device [1043:8677]
    Kernel driver in use: r8169
    Kernel modules: r8169, r8168
    --
    05:01.0 Ethernet controller [0200]: Intel Corporation 82541PI Gigabit Ethernet Controller [8086:107c] (rev 05)
    Subsystem: Intel Corporation PRO/1000 GT Desktop Adapter [8086:1376]
    Kernel driver in use: e1000
    Kernel modules: e1000
    08:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 01)
    Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller [10ec:8168]
    Kernel driver in use: r8169
    Kernel modules: r8169, r8168
    09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 710B] [10de:128b] (rev a1)


    After rebooting the driver r8168 is not loaded. So going with previous methods I have added

    "blacklist r8169"


    to /etc/modprobe.d/clearos.conf and created NIC.conf in the same folder (644) containing:


    alias enp3s0 r8168
    alias enp8s0 r8168


    but this doesn't seem to work too. Now I'm having even more issues :-(
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, May 10 2018, 04:13 PM - #Permalink
    Resolved
    0 votes
    Do a search for "Ryzen redhat" or "ryzen el7" or "ryzen centos" and see what you come up with. I bumped into this but have not searched extensively.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, May 10 2018, 04:07 PM - #Permalink
    Resolved
    0 votes
    Unless you've done something about it, you probably also have issues with the RTL8111/8168/8411 and strongly recommend you install the kmod-r8168 and kmod-r8169 drivers then reboot.

    The IntelPro/1000 GT I think is an e1000 card. Can you give the output to:
    lspck -knn | grep Eth -A 3
    modinfo e1000
    If it is an e1000 card there are no later kmod drivers available. The 8.?.? versions of the drivers did not work with el7 so we are stuck with the ones supplied with the kernel, unless you want to compile your own directly from Intel.

    [edit]
    I do have one question, though. Are you sure the Ryzen5 is compatible with EL7? I am not sure. EL7 is often not too up to date with hardware support.
    [edit]
    The reply is currently minimized Show
Your Reply