Re: Support for Cluster of SMP (or workstation) in DragonFly

["Noah Yan" <yanyh@xxxxxxxxx>]

Matt,

Thanks very much for discussion. I would appreciate very much if we 
continue, although it is out-of what the current DragonFly project does. See 
below...

Noah Yan

"Matthew Dillon" <dillon@xxxxxxxxxxxxxxxxxxxx> wrote in message 
news:200410141722.i9EHM7KO005082@xxxxxxxxxxxxxxxxxxxxxxx
> :Thanks very much for your answering, Matt
> :
> :...
> :are installed and should be tightly integrated with the kernel (but not 
> the
> :case). For scalability, it is very important to optimize global
> :synchornization and co-scheduling for parallel applications whose 
> processes
> :may be spawned from hundreds or tens of thousands. I am very intestered 
> in
> :...
> :
> :one more question(just for discussion, and this may more suitable for the
> :kernel group): is it posibble to organize the os in three level,
> :kernel-level, system-level, user-level(like Minix)? system level is
> :instroduced to address the issues from OS clustering. And all 
> context-switch
> :(or system call) to kernel-level are heavy, but user-system levels are
> :lightweight switch. So in clustering, all cross-box operation(mostly
> :messaging passing) happens between system-level with lightweith context
> :swtich. In this way, we can reduce the performance overhead by removing 
> the
> :context swtich cost to kernel level. System level is used perform basic
> :protection, which is less costly compared with kernel-level protection, 
> and
> :also different from user-land operation without any protection.
> :
> :Thanks very much
> :Noah Yan
>
>    I don't think it is possible to organize the OS in a strict three-level
>    hierarchy, at least not have anything efficient when you are done.  The
>    primarily issue is that you can't do cache coherency 'in a vacuum'. 
> That
>    is, for cache coherency to work efficiently, *all* the layers have to
>    know about it and *all* the layers have to have the ability to
>    pro-actively manipulate cache coherency states.  Thus, even though you
>    may have a distinct cache-coherency layer in the OS, the layer isn't
>    isolated.
For the cache coherency, I donot have experience in OS coding for them. what 
I understand is that it exist in a SMP box, and for cluster connected with 
network, no such issues. Am I right?
>
>    Also, each major subsystem has different interaction requirements in a
>    clustered system.  There is no clean way to separate the required
>    functionality out.
>
>    So rather then having distinct layers you instead have lots of 
> subsystem
>    pieces integrated together into a whole.  It isn't a jumble, but it 
> isn't
>    distinctly separateable into three layers either.
To organize them in to three level is not purely for functionalities, 
instead, mainly for protection. It is not historical hierarchical 
architecture of OS design(by putting process management, memory management, 
FS, etc on different levels). The key point of it is to reduce the cost of 
protection and context switch as much as posibble. Process's VM can be 
separated in three spaces(user, system, kernel). Kernel-level is for the 
protection for kernel memory, it is SMP-aware and handle cache coherency 
issues in a SMP system. System-level is cluster aware and responsible for 
the memory protection of system-level memory. Unlike the traditional 
hierarchy OS, each level has to deal with process, VM, driver management, 
and it is the root to decide where to put them for efficency and the context 
switch between user and system level must be lightwigtht to take advantage 
of the system-level.

One question you may ask ( I also ask myself) is "why not do this just in 
user land?" As far as effeciency is concerned,I donot have a convincing 
answer for it either because there is even no context-switch if just do it 
in user-land. But if that, the unavoided context-switch have to be 
user-kernel, which is costly.

Another question for it is "what are the memory area that do not require 
kernel protection, but require system protection". The simple answer for it 
is "system-level" space in process VM space. if that, some more 
investigation are needed regarding to adding one more space in current 
user-kernel VM space of a process.

Also, "how to do a lightweight context switch"


>
> -Matt
> Matthew Dillon
> <dillon@xxxxxxxxxxxxx>