kqueue selective wakeup mechanism

["Samuel J. Greear" <sjg@xxxxxxxxxxxx>]

As part of my Summer of Code work I have been looking at implementing
selective wakeups in one form or another. Currently (and probably
since the beginning of time) when some processes or threads are
waiting on a descriptor in one of select/poll/kevent, all waiting
processes/threads are woken up  when it is available, and that
functionality is now mandatory.

Quite a few ideas were considered but the only one that really makes
sense right now is an opt-in mechanism in the form of a kqueue flag,
EV_WAKEONE. The idea here is to be able to simplify a userspace
server, such as a webserver, to allow it to reasonably block in
kevent(2) and cut out a lot of the need or desire to do descriptor
passing and and cross-process/thread synchronization around kevent(2)
or accept(2). While at the same time improving performance, we aren't
needlessly waking a large number of processes or threads.

I looked at a large number of webservers in an attempt to find one
that operated in roughly this manner, but to no avail. So to perform
an adequate test I wrote one (well, not really, its the minimum I
could get away with and have apachebench come away happy):
http://pastie.org/1074861 - This server forks n processes (see for
loop) and all of them block in kevent waiting for new connections and
readiness of existing connections. This server isn't really useful
beyond the scope of this test, it does no parsing and effectively
relies on the socket buffers working ideally, but it should be
sufficient to illustrate the performance differences between waking up
n listeners and waking up 1.

Here http://pastie.org/1074873 is the output of apachebench run
against the test server in two modes, the first used the traditional
kqueue flags and races to accept(2), ignoring errors there. The second
used EV_WAKEONE and reported accept(2) errors (there were none). As
you can see the second test was approximately 16% faster on a 2 core
SMP test machine.

For reference, I have the results of lighttpd and apache1.3 subjected
to a nearly identical workload on the same machine
http://pastie.org/1074889

The kernel implementation of this functionality is rather simple:
http://pastie.org/1074878

Any thoughts on any of this would be appreciated.

Best,
Sam