Re: PF & SMP

[Matthew Dillon <dillon@xxxxxxxxxxxxxxxxxxxx>]

:How much time is really wasted where one cpu is doing nothing but
:waiting on one kthread to complete nowadays?
:
://jonas

    It only effects parallelism for cpu-heavy operations within the kernel.
    So, for example, a computer doing nothing but routing packets between
    two interfaces would be effected and a computer doing very tcp-heavy
    web server operations from data cached in memory would be effected
    but a computer doing user-centric stuff, such as serving mail or
    running a web-server with a heavy-weight backend (CGIs, java,
    database accesses), or doing lots of disk I/O, would not be effected
    as much.

    You can monitor big giant lock contention by observing the
    'lwkt.mplock_contention_count' sysctl.  This counter is incremented
    every time the LWKT scheduler has to spin due to wanting to schedule
    a thread that requires the MP lock but not being able to obtain the
    MP lock because another cpu is holding it.  Each tick of the counter
    represents an amount of time wasted that is based on the speed of the
    cpu.  I would guess around 100ns per tick on a modern cpu.

    As part of my parallel buildworld test I record the contention
    count delta for each -j 8 build.  I typically get numbers in the
    800-900 million range.  x 100 ns pertick == about 100 seconds of wasted
    of cpu (for both cpus together) for a buildworld that takes 1450 seconds,
    and dividing by the two cpus, that probably represents a real-world
    loss in build time of between 4% and 7% over what it could hav achieved.
    That is a very rough estimate, I haven't actually measured what a 'tick'
    is... it could be less then 100ns but I don't think it is more.

    A kernel-centric test, such as someone blasting TCP between two machines
    but not otherwise doing any userland processing of the data, will result
    in far greater contention.  Such tests are a good way to test potential
    contention, but don't really represent real-world effects. 

    The biggest effect MP contention has in a real-world system is 
    probably related to applications that communicate over local pipes
    or unix domain sockets... for example, a web server with a MYSQL backend.
    In this case being able to make the pipe and localhost socket code
    MP safe would reap large rewards vs making the rest of the kernel MP safe.
    Making thing like the disk I/O path MP safe doesn't generaly reap large
    rewards because the disk I/O itself, rather than the cpu, represents the
    bottleneck.

						-Matt