Re: userapi: signals

[Matthew Dillon <dillon@xxxxxxxxxxxxxxxxxxxx>]

:>     never makes sense to have more then NCPU kernel threads dedicated 
:>     to a user program, regardless of how many userland threads that 
:>     user program is running.
:
:OK, but that's what it's like in the kernel as well, isn't it? Kernel 
:threads are a lightweight per-cpu concept, really, with synchronization 
:between them being deferred to the more expensive IPI token passing...
:
:So...

    You mean in regards to multi-threading things like VFS devices?  Right,
    one shouldn't need more then NCPU threads per (high performance) device
    to maximize performance.

    I'm not sure how much kernel threading applies to user threading.  Kernel
    threads are more sensitive to memory issues since there is only one 
    KVM address space and very little of it is pageable.

:Maybe I'm confused or I'm not presenting myself well. Let me see if I 
:can get this straight. We can draw some parallels...
:
:	1a. Each cpu in the system has its own self-contained LWKT scheduler.

    Right.

:	1b. Each process in user mode has its own self-contained userland 
:thread scheduler.

    Right.  In *user mode* each process will be responsible for managing
    its own threads.  But a process *CAN* rfork() to take advantage of
    other cpus if it wishes.

:	2a. A Thread will never be moved to another cpu while it is running, 
:and a thread will never be preemptive switched to a non-interrupt thread.

    Right.

:	2b. A userland thread will never be moved to another CPU, and will 
:never be preempted by a non-signal thread.

    No.  In userland anything can happen.  The system is free to interrupt
    a userland program and migrate it to another cpu at any time.

    As an abstraction it should not matter.  The userland thread manager's
    concept of a 'cpu' is rfork().  It is up to the userland thread scheduler
    to determine which process to schedule a userland thread on and it doesn't
    much matter *what* the kernel does with those processes.  There is nothing
    preventing you from rfork()ing four userland processes on a single cpu
    system, and nothing preventing you from rfork()ing two userland processes
    on a quad MP system.

:	3a. A Thread can access non-global information without having to obtain 
:any locks...

    A thread can access the per-cpu globaldata area without obtaining any
    locks, but may have to enter a critical section if the information being
    accessed can also be modified by an interrupt.

    A thread can access any data in its thread structure or its process
    structure without getting any locks, but it may have to enter a critical
    section to protect against IPIs from other cpus manipulating data in
    its thread/process.

    critical sections are cheap... basically just a counter.

:	3b. A userland thread can access non-global information without having 
:to make any system calls...

    The rfork()'d 'process' that a userland thread is running on is under the
    complete control of the userland thread scheduler (or will be when this
    baby is written!).  This means that you can do whatever you want... you
    can create a per-rfork memory area and access it like a kernel thread
    would access its own per-cpu area, without locking.

:	4a. A cpu which attempts to schedule a thread belonging to another cpu 
:will issue an IPI-based message to the target cpu to execute the operation.

    Correct.  Or anything belonging to another cpu.  For example, if I had
    a per-cpu cache and I wanted to free a piece of data to another cpu's
    per-cpu cache I would send an asynch IPI to the other cpu with the data.

    IPIs have high latency but low cpu wastage (if you send them asynch),
    and they become more efficient as you get more heavily loaded down because
    the message queue can build up.  We could also implement a passive-IPI
    that doesn't bother actually sending the IPI to the target cpu and
    handle low-priority requests at task-switch time or something like that.
    (I don't do that yet but I could).  i.e. IPI messaging is potentially
    VERY flexible.

:	4b. A userland thread which attempts to schedule a thread belonging to 
:another CPU will make a system call (or send a message through a 
:non-system kernel message port).

    To another abstracted cpu (rfork).  Yes, though perhaps all it really
    needs to do is write() to a communications pipe that goes to the target
    (abstracted) cpu.  Thus this sort of operation would become more
    efficient under heavier loads.

:None of this means the kernel needs to know about userland threads, it 
:just means that the userland thread code is acting like the kernel 
:thread code. So the userland process gets a signal and it schedules the 
:signal thread... my question is more along the lines of:

    Right.  In fact, we could probably compile most of the LWKT subsystem
    separately as part of the userland scheduler.  The LWKT subsystem is
    almost completely self contained.  The messaging system *IS* 
    completely self contained.

:Can a process request a signal to fire when a message arrives on one of 
:its kernel message ports, in which case it would be able to support 
:preemptive signals, or would it have to wait for the signal to arrive on 
:its reply port, in which case signal handlers (or the signal thread) 
:would have to be scheduled synchronously when a Wait() occurs.

    Probably.  There are lots of ways to do this.  Through brainstorming
    and discussion we can figure out the most efficient mechanism to use.

					-Matt
					Matthew Dillon 
					<dillon@xxxxxxxxxxxxx>