moblog

Howto: Enable Receive Packet Steering (RPS) on Linux 2.6.35

by · Published · Updated

We run a high bandwidth Tor exit node on a Gbit connection. Unfortunately, the NIC by our hoster doesn’t support MSI-X to distribute interrupt load across all cores. The latest linux kernel 2.6.35 adds a mechanism called Receive Packet Steering:

[quote]This patch implements software receive side packet steering (RPS). RPS distributes the load of received packet processing across multiple CPUs.
Problem statement: Protocol processing done in the NAPI context for received packets is serialized per device queue and becomes a bottleneck under high packet load. This substantially limits pps that can be achieved on a single queue NIC and provides no scaling with multiple cores. ([url=http://lwn.net/Articles/328339/]lwn.net: Software receive packet steering[/url])[/quote]
What took us a lot of time to figure out: /proc/interrupts still shows only CPU0 is used for NIC interrupt handling, even with RPS enabled. If you want to find out whether RPS is working, you have to look at /proc/softirqs instead (eg. with watch -n1 cat /proc/softirqs):

                CPU0       CPU1       CPU2       CPU3
HI:          0          0          0          0
TIMER:  480622794  476948579  460999919  467641124
NET_TX:   25311134   27075847   27513332   27307975     <-----
NET_RX: 1388399338 4191697027 1491556667  627387845     <-----
BLOCK:    4632803          3     315726         29
BLOCK_IOPOLL:          0          0          0          0
TASKLET:         21          4          8          2
SCHED:  154913375  158601463   97907175  200790209
HRTIMER:    1576760    2361409    1330088    1545921
RCU:  421549961  407634645  405460584  415147363

In our case, we had to specifically enable RPS:

# cat /sys/class/net/eth0/queues/rx-0/rps_cpus
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,
00000000,00000000,00000000,00000000,00000000,00000000,00000000
# echo f > /sys/class/net/eth0/queues/rx-0/rps_cpus
# cat /sys/class/net/eth0/queues/rx-0/rps_cpus
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,
00000000,00000000,00000000,00000000,00000000,00000000,0000000f

I have added the echo line to /etc/interfaces to set it on each boot ("up ...").

More Information

  • [url=http://lwn.net/Articles/361440/]lwn.net: rps: Receive packet steering[/url]
  • [url=http://groups.google.com/group/redis-db/msg/cf5c4932af5662c5]Google Group Redis-DB: better multi-core functionality via per-process shared memory queues[/url]