IP forwarding performance

[Sepherosa Ziehau <sepherosa@xxxxxxxxx>]

Hi all,

This email serves as the base performance measurement for further
network stack optimization (as of git 107282b).


The hardware:
mobo ASUS P867H-M
4x4G DDR3 memory
CPU i7-2600 (w/ HT and Turbo Boost enabled, 4C/8T)
Forwarding NIC Intel 82576EB dual copper
Packet generator NICs Intel 82571EB dual copper


A emx1 <---> igb0 forwarder igb1 <---> emx1 B

A and "forwarder", B and "forwarder" are directly connected using CAT6 cables.
Polling(4) is enabled on igb1 and igb0 on "forwarder".  Following
tunables are in /boot/loader.conf:
kern.ipc.nmbclusters="524288"
net.ifpoll.user_frac="10"
net.ifpoll.status_frac="1000"
Following sysctl is changed before putting igb1 into polling mode:
sysctl hw.igb1.npoll_txoff=4


First for the users that are only interested in the bulk forwarding
performance:  The 32 netperf TCP_STREAMs running on A could do
941Mbps.


Now the tiny packets forwarding performance:

A and B generate 18 bytes UDP datagrams using
tools/tools/netrate/pktgen.  The destination addresses of the UDP
datagrams are selected that the generated UDP datagrams will be evenly
distributed the to the 8 RX queues, which should be common in the
production environment.

Bidirectional normal IP forwarding:
1.42Mpps in each direction, so total 2.84Mpps are forwarded.
CPU usage:
On CPUs that are doing TX in addition to RX: 85% ~ 90% (max allowed by
polling's user_frac)
On CPUs that are only doing RX: 40% ~ 50%

Bidirectional fast IP forwarding: (net.inet.ip.fastforwarding=1)
1.48Mpps in each direction, so total 2.96Mpps are forwarded.
CPU usage:
On CPUs that are doing TX in addition to RX: 65% ~ 70%
On CPUs that are doing RX: 30% ~ 40%

Best Regards,
sephe

--
Tomorrow Will Never Die