Traffic shaping

This article aims to give a basic foundation to start traffic shaping to improve responsiveness (ping, RTT) on internet links. Especially asynchronous links like DSL benefit from this. PING round trip time can be improved as much as 10x during heavy download/upload with this traffic shaping in place.

Prerequisites
You need to configure your Kernel with QoS support. You should add HTB, FQ_CODEL and Ingress queuing disciplines as well as IFB (Intermediate Functional Block device) and U32 match support.

Kernel
You need to activate the following kernel options:

If you have IPv6 support on your network or plan to use IPv6 tunnels, you may set IPv6 Protocol to . You should also enable Netfilter options, which enables the use of iptables firewall.

Iproute2
Install :

The iproute2 package installs the powerful tool called "tc". It is used to control and modify queuing and filters on network links.

Lets begin with a little example:

By default, the pfifo_fast queuing discipline is used by the Linux kernel. It should not be listed by "tc qdisc show".

Theory
There are two modes of traffic shaping, INGRESS and EGRESS. INGRESS handles incoming traffic and EGRESS outgoing traffic. Linux does not support shaping/queuing on INGRESS, but only policing. Therefore IFB exists, which we can attach to the INGRESS queue while we can add any normal queuing like FQ_CODEL as EGRESS queue on the IFB device.

The main reason for traffic shaping is that we cannot control the packet queues or prioritisation made by our ISP or in the external link. By limit our maximum bandwidth to 90% of our link speed we can make sure that any buffers (queues) that our ISP and our external link has will remain empty.

FQ_CODEL is a queuing discipline that is based on AQM (Active Queue Management). It aims to create fair bandwidth for all flows, while attempting to minimise buffers (and hence delays).

The INGRESS shaping below works like this:
 * 1) Create ingress filter on external interface
 * 2) Copy all incoming data to the IFB device
 * 3) Create an EGRESS qdisc on the IFB device and limit the bandwidth to 90%
 * 4) Attach the FQ_CODEL queuing discipline.

Note
The original version of this page was wildly incorrect, in particular the ingress portion of the shaper didn't work at all. The protocol filter was ip only (no arp, no ipv6), the ifb0 device was not up, the default htb destination was to the direct, not fq_codel qdisc. Also the htb "quantum" is different and has a different purpose than the the fq_codel quantum - the htb quantum is there to lighten the cpu load of htb under higher rates, the fq_codel quantum is there to optimize for different packet sizes....

It was good to know everything that can go wrong on a fq_codel rate shaper... -- dtaht

I generally recomend treating the ecn idea gently on anything but strictly controlled networks (like data centers)

Traffic Shaping Script
It is important that you start by setting your upload and download speeds to about 90% of your maximum link speed. After you get satisfying results, you can generally try increasing your upload speed to 95% or higher, and twiddle with download speed. On some links you can get away with 95% too, on some, 85% is safer. ADSL has special framing problems which require usage of the htb STAB parameter (not shown here).

You also need to load all modules beforehand.

Statistics
To see if the new shaping is activated you should run some heavy traffic through it, and check your work. This example uses netperf. It is helpful to run multiple instances in both directions to really exercise things.....

ping -c 25 -H 172.21.2.1 & # watch your ping times go by. Should generally only increase by 10ms max root@ida:~/gen# netperf -H 172.21.2.1 -t TCP_MAERTS MIGRATED TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.21.2.1 port 0 AF_INET : demo Recv  Send    Send Socket Socket Message  Elapsed Size  Size    Size     Time     Throughput bytes bytes   bytes    secs. 10^6bits/sec

87380 65536  65536    10.00       6.78   root@ida:~/gen# netperf -H 172.21.2.1 -t TCP_STREAM MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 172.21.2.1 port 0 AF_INET : demo Recv  Send    Send Socket Socket Message  Elapsed Size  Size    Size     Time     Throughput bytes bytes   bytes    secs. 10^6bits/sec

87380 16384  16384    10.16       0.75

Then you can check your work:

qdisc htb 1: root refcnt 2 r2q 10 default 11 direct_packets_stat 0 Sent 4211186 bytes 16165 pkt (dropped 0, overlimits 10147 requeues 0) backlog 0b 0p requeues 0 qdisc fq_codel 801a: parent 1:11 limit 10240p flows 1024 quantum 300 target 5.0ms interval 100.0ms Sent 4211186 bytes 16165 pkt (dropped 343, overlimits 0 requeues 0) backlog 0b 0p requeues 0 maxpacket 1514 drop_overlimit 0 new_flow_count 2234 ecn_mark 0 new_flows_len 0 old_flows_len 18 qdisc ingress ffff: parent ffff:fff1 Sent 24181003 bytes 19329 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0

root@ida:~/gen# tc -s filter show dev eth0 parent ffff: filter protocol all pref 49152 u32 filter protocol all pref 49152 u32 fh 800: ht divisor 1 filter protocol all pref 49152 u32 fh 800::800 order 2048 key ht 800 bkt 0 terminal flowid ??? match 00000000/00000000 at 0 action order 1: mirred (Egress Redirect to device ifb0) stolen index 13 ref 1 bind 1 installed 517 sec used 0 sec Action statistics: Sent 34015624 bytes 29704 pkt (dropped 0, overlimits 0 requeues 0) backlog 0b 0p requeues 0

qdisc htb 1: root refcnt 2 r2q 10 default 11 direct_packets_stat 1 Sent 23852231 bytes 19336 pkt (dropped 0, overlimits 25453 requeues 0) backlog 0b 0p requeues 0 qdisc fq_codel 8019: parent 1:11 limit 10240p flows 1024 quantum 300 target 5.0ms interval 100.0ms ecn Sent 23852120 bytes 19335 pkt (dropped 463, overlimits 0 requeues 0) backlog 0b 0p requeues 0 maxpacket 1514 drop_overlimit 0 new_flow_count 2170 ecn_mark 0 new_flows_len 1 old_flows_len 1

Some notes
If you see a maxpacket greater than 1514, you have some tso/gso/gro offload on on some interface. If you don't see ecn_marking, one side or another of your hosts doesn't have ecn on, or it's disabled as per this script.

Excplicit Congestion Notification (ECN)
You can enable ECN (Explicit Congestion Notification) since FQ_CODEL will this to notify the internet hosts about congestions (bandwidth over limit), without actually dropping the packet.

However usage of ecn over the wide internet is kind of dubious, so this script only enables it on ingress, and in these examples, ecn_mark=0 as it is not enabled on the other side of the tested connection.

External resources

 * information about FQ_CODEL qdisc
 * iptables and stateful firewall
 * iptables and stateful firewall
 * Van Jacobson and others on codel and bufferbloat