git: kernel - Optimize kfree() to greatly reduce IPI traffic

[Matthew Dillon <dillon@xxxxxxxxxxxxxxxxxxxxxxx>]

commit 5fee07e60dc4c041779e46c199f144a7f7c550ee
Author: Matthew Dillon <dillon@apollo.backplane.com>
Date:   Sat Sep 18 13:23:41 2010 -0700

    kernel - Optimize kfree() to greatly reduce IPI traffic
    
    * Instead of IPIing the chunk being freed to the originating cpu we
      use atomic ops to directly link the chunk onto the target slab.
      We then notify the target cpu via an IPI message only in the case where
      we believe the slab has to be entered back onto the target cpu's
      ZoneAry.
    
      This reduces the IPI messaging load by a factor of 100x or more.
      kfree() sends virtually no IPIs any more.
    
    * Move malloc_type accounting to the cpu issuing the kmalloc or kfree
      (kfree used to forward the accounting to the target cpu).  The
      accounting is done using the per-cpu malloc_type accounting array
      so large deltas will likely accumulate, but they should all cancel
      out properly in the summation.
    
    * Use the kmemusage array and kup->ku_pagecnt to track whether a
      SLAB is active or not, which allows the handler for the asynchronous IPI
      to validate that the SLAB still exists before trying to access it.
    
      This is necessary because once the cpu doing the kfree() successfully
      links the chunk into z_RChunks, the target slab can get ripped out
      from under it by the owning cpu.
    
    * The special cpu-competing linked list is different from the linked list
      normally used to find free chunks, so the localized code and the
      MP code is segregated.
    
      We pay special attention to list ordering to try to avoid unnecessary
      cache mastership changes, though it should be noted that the c_Next
      link field in the chunk creates an issue no matter what we do.
    
      A 100% lockless algorithm is used.  atomic_cmpset_ptr() is used
      to manage the z_RChunks singly-linked list.
    
    * Remove the page localization code for now.  For the life of the
      typically chunk of memory I don't think this provided much of
      an advantage.
    
    Prodded-by: Venkatesh Srinivas

Summary of changes:
 sys/kern/kern_slaballoc.c |  483 ++++++++++++++++++++++++++++++---------------
 sys/sys/malloc.h          |    5 +-
 sys/sys/slaballoc.h       |   12 +-
 3 files changed, 338 insertions(+), 162 deletions(-)

http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/5fee07e60dc4c041779e46c199f144a7f7c550ee


-- 
DragonFly BSD source repository