Re: cvs commit: src/sys/dev/netif/nfe if_nfe.c if_nfevar.h

["Sepherosa Ziehau" <sepherosa@xxxxxxxxx>]

On Wed, Jul 9, 2008 at 6:27 PM, Michael Neumann <mneumann@ntecs.de> wrote:
> Sepherosa Ziehau wrote:
>>
>> sephe       2008/07/05 00:29:44 PDT
>>
>> DragonFly src repository
>>
>>  Modified files:
>>    sys/dev/netif/nfe    if_nfe.c if_nfevar.h  Log:
>>  Use hardware timer to simulate interrupt moderation.
>>    hw.nfeX.imtimer -> 0  no interrupt moderation at all.
>
> With this setting I get:
>
>  intr 10 at 40001/40000 hz, livelocked limit engaged!
>  intr 10 at 3258/20000 hz, livelock removed
>
> and transfer speed of around 30 MB/sec.
>
>>  hw.nfeX.imtimer -> -Y hardware timer simulated interrupt moderation,
>>                        the simultated interrupt moderation timer will
>>                        be set to Y.
>
> A value of -125 here increases transfer speed to >90 MB/sec and shows no
> livelocks.

When I use 64K window to do TCP_STREAM TX/RX tests, I usually get
933Mbps, with imtimer to -125.  It looks like you are doing TX on dfly
box.  I have some vague ideas that "semi" turn off TX interrupts may
work and _may_ improve TX performance for small packets (have not
measured with pktgen yet); it may also fixes the watchdog on certain
hardware.

>
> Note that this is measured transfering a big file from hammer which is
> saved to /dev/null on the client side as otherwise my laptops hard disk
> would be the limiting factor.
>
> A value of 92-97 MB/sec is pretty good I think (for ftp). I'm looking
> for a better way to test the throughput of the box (I don't want to be
> the servers hard-disk the limiting factor). Any recommendations? I tried

netperf or iperf.  To measure pure TX performance of driver, you could
use pktgen in our tools/tools/netrate.

> netcat but that doesn't seem to work. Note that my client is FreeBSD
> (and network interface bge).

If your bge(4) is PCI one, you probably won't get more out of it;
700Mbps is pretty good.
BTW, I remembered bde@freebsd.org had posted a patch to tune interrupt
moderation for freebsd's bge(4), you may try that patch.  The default
setting in freebsd's bge(4) is not very good.  You may consider
changing bge_{rx,tx}_max_coal_bds to higher values.

>
> Could I expect an even higher throughput when setting hw.nfeX.imtimer to
> -250 (together with changing the rx_ring_count)?

Higher im is useful for small packets TX/RX, especially during packet
forwarding; too high value probably will not help host bulk data xfer,
with higher im, you will need larger TCP window, like 128K.

>
> Thanks for this great work!

Thank you for testing it.  I probably will change nfe(4)'s imtimer
from 0 to -125 after this release.

Best Regards,
sephe

-- 
Live Free or Die