Re: Interactive performance regression since 2.0 (patch #2)

[Matthew Dillon <dillon@xxxxxxxxxxxxxxxxxxxx>]

    Ok, here's a new patch to try:

	fetch http://apollo.backplane.com/DFlyMisc/sched02.patch

    This turned into a three-day marathon run but I'm really happy with
    the results.

    The old scheduler had a huge number of issues.  I can't count the number
    of issues I came across.  I had to substantially rewrite the logic.

    The new user scheduler does not depend on the helper threads when
    switching under load.  It uses a very cool algorithm whereby threads
    trying to return back into user mode 'bid' for the single user mode
    slot (per-cpu).  So if thread A bids the best priority so far,
    then switches to thread B (also trying to exit to user mode) which
    bids a better priority, thread B will directly deschedule thread A
    and take over the bid.  The scheduler helper is only used to kick
    idle cpus to pick up the threads that lost the bid.  This results in
    highly optimal operation.

    I also found an issue with MP lock contention on SMP boxes.  If the
    LWKT scheduler could not acquire the MP lock it would continue looking
    for threads that didn't need it.  Sounds great, but what was happening
    was that the scheduler was skipping important kernel threads needing
    the lock and scheduling cpu-intensive user threads, and then not
    rescheduling for a whole tick.  I created two fixes for this issue
    controllable live with a sysctl.

    lwkt.chain_mplock=0		Causes the LWKT scheduler to refuse to
				schedule a user thread if kernel threads
				exist which need the MP lock but couldn't
				get it.

    lwkt.chain_mplock=1		Causes the LWKT scheduler to schedule the
				user thread but adds logic to rel_mplock()
				to attempt to IPI some other cpu needing
				the MP lock, creating an event that allows
				that other cpu to reschedule out of the user
				thread.

    I intend to commit this on Friday baring problems, so please test both
    modes.

    So far on my desktop I have tested lwkt.chain_mplock=0 with excellent
    results.

					-Matt
					Matthew Dillon 
					<dillon@backplane.com>