Greetings,
First a brief summary of my ClearOS system:
ClearOS 7.5.0, 3.10.0-862.11.6.v7.x86_64 #1 SMP Wed Aug 15 20:03:47 MDT 2018
8 CPU(core) Xeon E3-1260L v5 2.90Ghz, 32GB RAM, 8 Ethernet Intel I210 ports, igb driver ver 5.4.0-k
Network topology:
WAN network and ISP gw----[ClearOS FW]---Internal network
Should scream, right? Yeah not so much....
I've been seeing bad inbound "download" performance so I started using iperf3 on different network segments to find the bad and debug it. Running iperf3 from the ClearOS box to machines on the internal network runs great, ~930-960Mbit in both directions.
Running the same test on the ClearOS box to a machine on my WAN network the outbound performance is great, inbound is terribly broken.
iperf3 send from ClearOS firewall to WAN system (my server on local WAN network)
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 114 MBytes 956 Mbits/sec 0 542 KBytes
[ 4] 1.00-2.00 sec 112 MBytes 938 Mbits/sec 0 542 KBytes
[ 4] 2.00-3.00 sec 112 MBytes 938 Mbits/sec 0 542 KBytes
[ 4] 3.00-4.00 sec 113 MBytes 948 Mbits/sec 0 542 KBytes
[ 4] 4.00-5.00 sec 112 MBytes 938 Mbits/sec 0 542 KBytes
[ 4] 5.00-6.00 sec 112 MBytes 938 Mbits/sec 0 542 KBytes
[ 4] 6.00-7.00 sec 113 MBytes 948 Mbits/sec 0 542 KBytes
[ 4] 7.00-8.00 sec 112 MBytes 938 Mbits/sec 0 542 KBytes
[ 4] 8.00-9.00 sec 112 MBytes 939 Mbits/sec 0 542 KBytes
[ 4] 9.00-10.00 sec 112 MBytes 938 Mbits/sec 0 542 KBytes
iperf3 send from WAN system to ClearOS firewall
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 10.4 MBytes 87.4 Mbits/sec
[ 4] 1.00-2.00 sec 7.39 MBytes 62.0 Mbits/sec
[ 4] 2.00-3.00 sec 90.4 MBytes 758 Mbits/sec
[ 4] 3.00-4.00 sec 107 MBytes 901 Mbits/sec
[ 4] 4.00-5.00 sec 1.93 MBytes 16.2 Mbits/sec
[ 4] 5.00-6.00 sec 2.36 MBytes 19.8 Mbits/sec
[ 4] 6.00-7.00 sec 3.91 MBytes 32.8 Mbits/sec
[ 4] 7.00-8.00 sec 1.99 MBytes 16.7 Mbits/sec
[ 4] 8.00-9.00 sec 2.17 MBytes 18.2 Mbits/sec
[ 4] 9.00-10.00 sec 2.24 MBytes 18.8 Mbits/sec
This is awful. I should see near same performance each direction. How do I resolve this? <100MBit? It's a gigabit interface connected to a gigabit switch connected to another local machine on the WAN network with a gigabit interface.
ClearOS is doing something wrong/bad/dumb. I looked at Bandwidth and QoS Manager and disabled the engine but there is no change in the terrible inbound performance. The WAN interface is Automatic/Automatic for Rate-to-Quantum.
Running top on the ClearOS box all I see creating a load is snort and that hovers in the 25-35% range during the iperf3 tests. No iowait, no swap, nothing obviously wrong.
tuned-adm is set to balanced, I assume this is a choice made by ClearOS developers. Latency-performance seems a better fit but I'm not going to start turning knobs in the OS since it is supposed to be an appliance-like product.
What can I do to get the inbound traffic on the WAN interface to perform at the same levels as outbound?
Thanks,
--Jeff
First a brief summary of my ClearOS system:
ClearOS 7.5.0, 3.10.0-862.11.6.v7.x86_64 #1 SMP Wed Aug 15 20:03:47 MDT 2018
8 CPU(core) Xeon E3-1260L v5 2.90Ghz, 32GB RAM, 8 Ethernet Intel I210 ports, igb driver ver 5.4.0-k
Network topology:
WAN network and ISP gw----[ClearOS FW]---Internal network
Should scream, right? Yeah not so much....
I've been seeing bad inbound "download" performance so I started using iperf3 on different network segments to find the bad and debug it. Running iperf3 from the ClearOS box to machines on the internal network runs great, ~930-960Mbit in both directions.
Running the same test on the ClearOS box to a machine on my WAN network the outbound performance is great, inbound is terribly broken.
iperf3 send from ClearOS firewall to WAN system (my server on local WAN network)
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 114 MBytes 956 Mbits/sec 0 542 KBytes
[ 4] 1.00-2.00 sec 112 MBytes 938 Mbits/sec 0 542 KBytes
[ 4] 2.00-3.00 sec 112 MBytes 938 Mbits/sec 0 542 KBytes
[ 4] 3.00-4.00 sec 113 MBytes 948 Mbits/sec 0 542 KBytes
[ 4] 4.00-5.00 sec 112 MBytes 938 Mbits/sec 0 542 KBytes
[ 4] 5.00-6.00 sec 112 MBytes 938 Mbits/sec 0 542 KBytes
[ 4] 6.00-7.00 sec 113 MBytes 948 Mbits/sec 0 542 KBytes
[ 4] 7.00-8.00 sec 112 MBytes 938 Mbits/sec 0 542 KBytes
[ 4] 8.00-9.00 sec 112 MBytes 939 Mbits/sec 0 542 KBytes
[ 4] 9.00-10.00 sec 112 MBytes 938 Mbits/sec 0 542 KBytes
iperf3 send from WAN system to ClearOS firewall
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 10.4 MBytes 87.4 Mbits/sec
[ 4] 1.00-2.00 sec 7.39 MBytes 62.0 Mbits/sec
[ 4] 2.00-3.00 sec 90.4 MBytes 758 Mbits/sec
[ 4] 3.00-4.00 sec 107 MBytes 901 Mbits/sec
[ 4] 4.00-5.00 sec 1.93 MBytes 16.2 Mbits/sec
[ 4] 5.00-6.00 sec 2.36 MBytes 19.8 Mbits/sec
[ 4] 6.00-7.00 sec 3.91 MBytes 32.8 Mbits/sec
[ 4] 7.00-8.00 sec 1.99 MBytes 16.7 Mbits/sec
[ 4] 8.00-9.00 sec 2.17 MBytes 18.2 Mbits/sec
[ 4] 9.00-10.00 sec 2.24 MBytes 18.8 Mbits/sec
This is awful. I should see near same performance each direction. How do I resolve this? <100MBit? It's a gigabit interface connected to a gigabit switch connected to another local machine on the WAN network with a gigabit interface.
ClearOS is doing something wrong/bad/dumb. I looked at Bandwidth and QoS Manager and disabled the engine but there is no change in the terrible inbound performance. The WAN interface is Automatic/Automatic for Rate-to-Quantum.
Running top on the ClearOS box all I see creating a load is snort and that hovers in the 25-35% range during the iperf3 tests. No iowait, no swap, nothing obviously wrong.
tuned-adm is set to balanced, I assume this is a choice made by ClearOS developers. Latency-performance seems a better fit but I'm not going to start turning knobs in the OS since it is supposed to be an appliance-like product.
What can I do to get the inbound traffic on the WAN interface to perform at the same levels as outbound?
Thanks,
--Jeff
Share this post:
Responses (10)
-
Accepted Answer
Usual culprits are:
1 - RTL8111/8168/8411 NIC. You don't have one
2 - Proxy/Disk speed
3 - IDS/IPS
4 - Bandwidth/QoS
5 - PPPoE on the WAN
2 - Are you running the proxy? If so, you will probably need to switch it to cacheless mode. There is a Howto covering that.
3 - Try disabling both for a start. If that works, make sure you are only running rules that you have for services you have exposed to the internet e.g. if you are not running the mail stack, there is no point in using the POP/IMAP/SMTP rules. Also snort (the IDS/IPS engine) makes a poor firewall, so rule sets like drop, botcc, compromised, dshield and tor are not very efficient at all. I use a script to convert them to proper firewall rules with using ipset sets.
4 - Make sure you have set your speed limits correctly. You can try disabling it to see if it makes an improvement. Also currently it uses IMQ devices in the kernel which requires some (horrible) kernel patching. This will be changing to use the built-in IFB devices, and ClearOS is ready for the change. If you want to try IFB there was a slightly messed up update in November which did not patch current systems. The top of /etc/clearos/qos.con should read:
QOS_ENABLE will be "on" if you are using it. The second section may be missing. Add it and change "no" to "yes". Then I am not sure if a firewall restart is sufficient or if a reboot is needed. I don't think a reboot is needed.# Enable ClearOS Bandwidth QoS Engine
QOS_ENABLE="off"
# Enable IFB over IMQ (yes/no)?
QOS_ENABLE_IFB="no"
5 - You seem to be doing an Ethernet<->Ethernet test so should be OK here. If you're using PPPoE externally, please say. -
Accepted Answer
Nick,
So i have taken what you put here, and i see that i have two of the same nics(RTL8111/8168/8411). Yesterday i had uverse fiber installed...
My speeds are pathetic...from uverse gateway 990down/994 up. The clearos server is 38 mgbs down/112 up at the moment.
I purchased this yesterday. Intel 8576
Think this could eleviate my speed issue? -
Accepted Answer
If you have those NIC's there may be an easy fix. Please do a:
this should install two files. Then reboot.yum install kmod-r816*
should then show the only available driver and the driver in use as r8168 (currently it will show r8169).lspci =k | grep Eth -A 3
[edit]
Your brief system summary said 8 i210 ports so I am a little confused. You could try testing with one of those ports
[/edit] -
Accepted Answer
Usual culprits are:
1 - RTL8111/8168/8411 NIC. You don't have one
Nope. Eight ports of Intel i210 PCIe Ethernet using igb driver which is known to be solid.
2 - Proxy/Disk speed
No proxy services are in use. Can't even find any proxy settings or references so those modules are likely not installed. Can you give me a clue of what I'm looking for to verify? All I have under Gateway is "Intrusion Protection".
3 - IDS/IPS
Wait, IDS/IPS suck? Why did I buy ClearOS then when I could just put up a skinny Linux box with locked down firewalld rulesets?
Result: Turning off IDS and IPS did not change the iperf3 benchmark results. Outbound from ClearOS goes wire speed, Inbound to ClearOS ~70Kbit.
4 - Bandwidth/QoS
/etc/clearos/qos.conf has QOS_ENABLE="off". There is no QOS_ENABLE_IFB entry.
5 - PPPoE on the WAN
Not using PPPOE. Just regular old Ethernet. ISP provides metro area Ethernet as a Cat5 cable and ipaddr/netmask. -
Accepted Answer
Nick,
Thanks for your response...i tried your suggestions and i am seeing the same result.
CLI Speed test from Clearos gateway
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Sumner Communications (Wellington, KS) [65.42 km]: 58.36 ms
Testing download speed................................................................................
Download: 36.34 Mbit/s
Testing upload speed................................................................................................
Upload: 90.75 Mbit/s
lspci -k | grep Eth -A 3 - After Reboot, and driver install
[root@ ~]# lspci -k | grep Eth -A 3
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
Subsystem: ASUSTeK Computer Inc. P8P67 and other motherboards
Kernel driver in use: r8168
Kernel modules: r8168
--
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
Subsystem: Realtek Semiconductor Co., Ltd. Device 0123
Kernel driver in use: r8168
Kernel modules: r8168
[root@ ~]#
Speedtest from the Uverse Gateway..
04/04/19 01:31:49 PM upstream 908.735 70 39.791
04/04/19 01:31:37 PM downstream 992.652 70 18.259
04/03/19 06:59:13 PM upstream 914.425 70 31.612
04/03/19 06:59:01 PM downstream 990.489 70 18.036
04/03/19 04:00:38 PM upstream 914.715 70 41.303
04/03/19 04:00:26 PM downstream 994.429 70 18.153
04/03/19 03:42:04 PM upstream 915.694 70 51.980
04/03/19 03:41:51 PM downstream 992.946 70 18.266
My comment earlier was about the intel nic i ordered from amazon. I was curious if changing the nic hardware would make a difference or not.
I don't have the QoS app installed all other ids/ips is disabled. I have set proxy to cacheless mode and might disable for testing.
From the gateway to uverse and vice versa...both connections are set to auto, and show to be connected at gigabit.
This problem is weird, which is why i think it might be hardware related...
I am on the community edition of clearos 7, with latest updates. -
Accepted Answer
I have completely UNINSTALLED IDS/IPS and QOS and Bandwidth Manager...after which I rebooted.
I re-ran iperf3 tests over the WAN interface, same garbage inbound performance.
In the below tests, the iperf3 server system is connected to the same gigabit switch as the ClearOS WAN port
[root@clearos ~]# iperf3 -c <redacted> -f m
Connecting to host <redacted>, port 5201
[ 4] local <redacted> port 35752 connected to <redacted> port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 115 MBytes 963 Mbits/sec 0 574 KBytes
[ 4] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 0 574 KBytes
[ 4] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 0 574 KBytes
[ 4] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 0 574 KBytes
[ 4] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 0 574 KBytes
[ 4] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 0 574 KBytes
[ 4] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 0 574 KBytes
[ 4] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 0 574 KBytes
[ 4] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 0 574 KBytes
[ 4] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 0 574 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver
iperf Done.
[root@clearos ~]# iperf3 -c <redacted> -f m -R
Connecting to host <redacted>, port 5201
Reverse mode, remote host <redacted> is sending
[ 4] local <redacted> port 35760 connected to <redacted> port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 9.25 MBytes 77.6 Mbits/sec
[ 4] 1.00-2.00 sec 3.11 MBytes 26.1 Mbits/sec
[ 4] 2.00-3.00 sec 3.17 MBytes 26.6 Mbits/sec
[ 4] 3.00-4.00 sec 3.29 MBytes 27.6 Mbits/sec
[ 4] 4.00-5.00 sec 2.98 MBytes 25.0 Mbits/sec
[ 4] 5.00-6.00 sec 3.48 MBytes 29.2 Mbits/sec
[ 4] 6.00-7.00 sec 3.11 MBytes 26.1 Mbits/sec
[ 4] 7.00-8.00 sec 3.23 MBytes 27.1 Mbits/sec
[ 4] 8.00-9.00 sec 3.29 MBytes 27.6 Mbits/sec
[ 4] 9.00-10.00 sec 2.98 MBytes 25.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 39.1 MBytes 32.8 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 37.9 MBytes 31.8 Mbits/sec receiver
iperf Done.
So I can send to the iperf3 server at near wire speed and back at less than 10% of that speed.
QOS/BandwithMgr and IDS/IPS are totally uninstalled and the ClearOS box rebooted. No Proxy. WHY IS THIS SO SLOW? -
Accepted Answer
I had not realised the thread had split.
@Joshua, I have no idea. What is the Uverse gateway? Is that in your house?
@Jeff,
2 - Would be under Gateway > Content Filter and Proxy so you don't have it.
3 - Yes it consumes a lot of resources but is bound to a single core. Suricata can use multiple cores but has other issues. If you didn't have a very high speed line ......
4 - So you are not using QoS. Feel free to add the other entries. An update will be pushed after 7.6 comes out to do this update.
5 - Fine. There would have been an alternative for PPPoE
Which leaves me struggling for ideas. What apps do you have installed:rpm -qa | grep ^app- | grep -v -e '-core' | sort
In network settings can you check you have not accidentally set an upstream proxy?
When you next do a speed test, please monitor the output of "top" to see for any excess usage. -
Accepted Answer
-
Accepted Answer
-
Accepted Answer
I have just tested from a ClearOS server on my LAN as a client and my ClearOS gateway as the iperf server and I get:
The client is just an HP Microserver with a dual core AMD X3216 processor and the server is a homebrew thing with an oldish 2 core/4 thread i3-4130 and a i210 NIC, so there must be something else at work giving slow figures.[root@microserver /]# iperf3 -c 172.17.2.1 -R -f m
Connecting to host 172.17.2.1, port 5201
Reverse mode, remote host 172.17.2.1 is sending
[ 4] local 172.17.2.5 port 43790 connected to 172.17.2.1 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 112 MBytes 940 Mbits/sec
[ 4] 1.00-2.00 sec 112 MBytes 941 Mbits/sec
[ 4] 2.00-3.00 sec 112 MBytes 938 Mbits/sec
[ 4] 3.00-4.00 sec 112 MBytes 941 Mbits/sec
[ 4] 4.00-5.00 sec 112 MBytes 941 Mbits/sec
[ 4] 5.00-6.00 sec 112 MBytes 941 Mbits/sec
[ 4] 6.00-7.00 sec 112 MBytes 941 Mbits/sec
[ 4] 7.00-8.00 sec 112 MBytes 939 Mbits/sec
[ 4] 8.00-9.00 sec 112 MBytes 941 Mbits/sec
[ 4] 9.00-10.00 sec 112 MBytes 941 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver
iperf Done.
[root@microserver /]# iperf3 -c 172.17.2.1 -f m
Connecting to host 172.17.2.1, port 5201
[ 4] local 172.17.2.5 port 43794 connected to 172.17.2.1 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 114 MBytes 959 Mbits/sec 0 395 KBytes
[ 4] 1.00-2.00 sec 112 MBytes 941 Mbits/sec 0 403 KBytes
[ 4] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 0 411 KBytes
[ 4] 3.00-4.00 sec 112 MBytes 943 Mbits/sec 0 421 KBytes
[ 4] 4.00-5.00 sec 112 MBytes 939 Mbits/sec 0 421 KBytes
[ 4] 5.00-6.00 sec 113 MBytes 944 Mbits/sec 0 421 KBytes
[ 4] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 0 440 KBytes
[ 4] 7.00-8.00 sec 112 MBytes 938 Mbits/sec 0 440 KBytes
[ 4] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 0 440 KBytes
[ 4] 9.00-10.00 sec 112 MBytes 938 Mbits/sec 0 440 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec receiver
iperf Done.
Do you have MultiWAN?
Do you use netifyd (Application and Protocol filters)?
Please login to post a reply
You will need to be logged in to be able to post a reply. Login using the form on the right or register an account if you are new here.
Register Here »