malloc(M_NOWAIT) issues.

[Matthew Dillon <dillon@xxxxxxxxxxxxxxxxxxxx>]

    An interesting thread (search for '[CHECKER]' in freebsd-hackers led
    me to review our use of M_NOWAIT in malloc().

    In a nutshell, most of the M_NOWAIT uses are incorrect.  In particular,
    much of the CAM code runs from an interrupt or a software interrupt
    and almost universally uses M_NOWAIT for its allocations, with dire
    consequences if/when allocations fail.

    I think I've come up with a solution!  The reason M_NOWAIT is used is
    primarily because an interrupt thread may be preempting a normal thread
    and cannot safely manipulate 'cache' pages in the VM page queues.  This
    is because reusing a cache page requires messing around with the 
    VM Object the cache page resides in.  So M_NOWAIT causes kmem_malloc()
    to only pull pages out of the VM 'free' page queue.  At the same time
    it is allowed to actually exhaust the free page queue whereas normal
    allocations are not allowed to completely exhaust the free page queue.

    But in DragonFly it is utterly trivial to (A) test to see if we are
    preempting someone and (B) to switch away and when we get scheduled again
    we will no longer be preempting anyone and can use a more exhaustive
    allocation algorithm.

    This means that in DragonFly we do not have to key interrupt-time 
    allocations off of M_NOWAIT.  Instead we can simply test to see if we
    are preempting someone and if we are only allocate from the VM page
    'free' queue... and if that allocation fails all we need to do is
    lwkt_switch() away, and when we get cpu back we will no longer be
    preempting anyone and can safely access the other queues.

    This in turn (whew) means that with very few changes to the low level
    kmem_malloc() code we can allow interrupt code to use M_WAITOK for
    something like 90% of the cases where it currently uses M_NOWAIT.

    I am going to add a couple of new M_ flags and start 'fixing' all the
    improper M_NOWAIT useages to something a whole lot safer.

					-Matt
					Matthew Dillon 
					<dillon@xxxxxxxxxxxxx>