Locking and Synchronization

Lockmgr

Lockmgr locks used to be heavy-weight locks but have since been optimized and will run almost as fast as spinlocks and faster than mtx locks in the critical path. Lockmgr locks block formally and will not be as fast as spin-locks for very short highly-contending bits of code. Lockmgr locks are the go-to lock for DragonFly and should be used if you are not sure what to use.

Internals

Uses atomic_cmpset*()
Both shared and exclusive locks are supported.
Locks are held across blocking conditions.
Exclusive priority over shared for SMP performance.
Always a procedure call but fast-paths operations internally.
Will formally block when acquiring (if LK_NOWAIT not used).
Can unblock and return an error on signal received.
Has many general purpose features.
Easy to debug.
Easy wakeup() semantics with numerous cpu management features.
Most portable.

Spinlocks

Kernel spinlocks are optimized for fast inlining but should generally only be used if tokens, lockmgr, or mtx locks are not appropriate. The most common use case is when you are holding a token but do not want to break its atomicy by potentially blocking on a deeper lock. It is far more difficult to debug issues with spinlocks than it is for blockable locks.

Internals

Non-recursive by design.
Both shared and exclusive spinlocks are supported.
Thread spins to wait, does not sleep or block.
Uses atomic_cmpset*(), fast-path is inlined.
Any held tokens will not be temporarily released.
Automatic critical section while a spinlock is held.
Exclusive spinlocks have priority over shared spinlocks to reduce SMP lag.
Should only be used for very short segments of code.
Should not wrap long or complex pieces of code or anything that can block.
Should not wrap kernel calls, even simple ones.
Nesting is not recommended.
Do not use unless you are very familiar with how spinlocks work as misuse can lead to a kernel deadlock.

LWKT serializing tokens

LWKT tokens are commonly used to loosely lock very broad, deep, or complex sections of code which might block at times. You can acquire tokens in any order without worrying about deadlocks, however any blocking condition (including blocking when acquiring additional tokens) will temporarily release ALL held tokens while the thread is blocked. Tokens have the property of serializing code execution for the token held while the thread is runnable.

Preemption by another thread (typically an interrupt thread) will not break the tokens you are holding. Preemption is still allowed... tokens do not enter a critical section. If the preempting thread tries to obtain the same token it will reschedule normally and return to the original thread.

The kernel thread scheduler will try to spin a bit on a token that cannot be acquired before giving up and blocking for real.

Internals

Uses atomic_cmpset*() internally
Spins in scheduler instead of blocking but counts as 'blocking' because scheduler is allowed to switch to other threads.
Easy to debug, all held tokens are tracked on a thread-by-thread basis.
Recursion is always allowed.
Both exclusive and shared tokens are supported
Shared tokens must be used with caution, mixed use on the same token in the same thread will panic.
Optimal for ping-ponging between two threads.

MTX

Mtx locks in DragonFly should not be confused with mtx locks in FreeBSD. Mtx locks in DragonFly are a very different beast. Generally these locks work similarly to lockmgr locks but provide two additional features which also make them heavier-weight in the face of contention.

Internals

Also uses atomic_cmpset*() internally.
Implements lock chaining internally, round-robin for exclusive locks.
Exclusive locks have priority over shared locks for SMP performance.
Always recursive.
The fast-path is inlined.
Easy to debug due to internal chaining structures.
Locks are held across blocking conditions.
Supports shared and exclusive locks and most features of lockmgr locks.
Supports callback-based asynchronous lock acquisition.
Supports lock aborts (for both synchronous and asynchronous).

MPLock

The mplock has been deprecated and should no longer be used. It is still used in a few non-critical places in DragonFly but has been removed from nearly all major and minor kernel subsystems.

Internals

API is a wrapper for mp_token

LWKT Messages

DragonFly implements a light weight message-and-port subsystem which is primarily used to shove packets around in bulk in the network subsystem and to cpu-localize network operations. The disk subsystem also uses LWKT messages to sequence disk probes and synchronize devfs.

Internals

Message passing and queueing.
End-point mechanics can be customized.
Very light-weight.

Critical Sections

Critical sections are a form of cpu-localized locking that prevents a thread from being preempted by another thread or an IPI or clock interrupt.

Critical sections do not protect a thread from threads or interrupts which might run on other cpus and should generally not be used to protect the frontend of a chipset driver from the backend (since the frontend can run on any cpu). Critical sections are primarily used to protect against IPIs, allowing cpu localized code to run very efficiently.

A critical section can be used if you want to avoid code latency but is not recommended for this purpose any more. A critical section cannot prevent SMIs from occurring and also do not prevent the actual hard interrupt from interrupting the cpu, it simply causes the interrupt to immediately return and delay execution until the critical section is exited.

Internals

Prevents preemption by an interrupt thread
Prevents preemption by an IPI or clock interrupt
Critical section holder can still block if desired, the critical section is simply set aside and then restored when the thread unblocks.

Condvars

Condvars are only used when porting code from FreeBSD and should not generally be used for native DragonFly code.

Internals

Generally used for simple interlocked event/wait loops, but tends to obfuscate what is happening.
Uses a spinlock internally.
Not inline-optimized.
Not optimized for heavy SMP contention.
Lockmgr locks are generally easier to understand and should be used instead.

Serializers

DragonFly currently implements a LWKT serializer abstraction which should not be used generally. This abstraction is heavily integrated into the network interface device driver subsystem.