Re: Backporting DFly patches to FreeBSD?

["Devon H. O'Dell" <dodell@xxxxxxxxxxxxxxx>]

On Sun, 2005-02-20 at 13:16 -0800, Matthew Dillon wrote:
> :I really don't think that's the way the cookie crumbles, as you state
> :it. FreeBSD has very different design goals and I believe the project
> :will stick to them. Again, as some developers have said, they would like
> :to see some of these changes stabilize before importing them into
> :FreeBSD.
> :
> :Plenty of people are interested to get some of the DragonFly BSD work
> :into FreeBSD. Sascha's syscons changes are especially interesting for
> :the FreeBSD project as they would be quite simple to import compared to
> :some of our other advancements. I don't think that interest is a lacked
> :factor in this equation; I think that developer time is. While the
> :project has a relatively large number of developers, the number of these
> :who actively contribute and understand the parts of the kernel that are
> :affected is limited to the few who are already busy with several other
> :projects.
> :
> :I would personally like to see some well-defined scalability tests
> :(perhaps in-kernel hooks would be interesting to examine performance of
> :some of our more obscure modifications) before I see any more stuff
> :stating that we perform better than FreeBSD on the same hardware.
> :
> :I'd also like to see some people work on backporting changes to FreeBSD
> :and less fingers pointed towards the developers who are actively working
> :on the FreeBSD project. These are people who have families, jobs and do
> :already actively contribute. It's unfair to ask these people to drop
> :what they're doing and import DragonFly's changes. It's also
> :impractical: one reason that DragonFly exists is to explore how these
> :changes actually work in practice.
> :
> :I could generate pages of comparisons that would make it quite clear
> :that these aren't always good ideas nor reasonable expectations, but I
> :think that's clear by now.
> :
> :Kind regards,
> :
> :Devon H. O'Dell
> 
>     I have to say that I disagree with this assessment of FreeBSD, and I
>     disagree with the assessment as to why some of the work hasn't been
>     backported.  First of all, the stability argument can't be right.
>     FreeBSD developers are committing all sorts of things to FreeBSD-5/6
>     that are not only unstable, but *seriously* unstable and some
>     of those issues, like the scheduler, have taken a year or more to fix
>     (if indeed one could call it fixed now, which I doubt).  DragonFly's
>     development isn't bug free, but the regimen is a lot tighter then FreeBSD
>     development these days.  Most of the mechanisms that are still 
>     backportable have been stable in DragonFly for over a year.  For most
>     of the rest FreeBSD has simply waited too long and the divergence has 
>     become too great for any single person, even a dedicated one, to backport
>     the more interesting work.  So, for example, IPI messaging should be
>     backported, no question about it, and it still can be.  It's an absolute
>     requirement for them to backport it and consolidate all the myrid 
>     ridiculously complex IPI mechanisms they currently have, but nobody is
>     doing it.  I consider that a serious management failure on FreeBSD's part.
>     There is certainly interest in doing some backporting, and for those
>     developers not being able to do it is nothing more then a time constraint,
>     but there is also a level of hostility to any backported work and a
>     level scrutiny that goes way beyond the scrutiny applied to native work.
>     Generally lots of 'its not proven' excuses go flying around pretty much
>     ignoring the fact that similar FreeBSD development itself is just as
>     unproven.  In many respects, I think the perforce model they are using
>     has resulted in even more isolation of sub projects within FreeBSD, and
>     it hasn't seemed to helped in the bug department for work that finally
>     gets into the CVS tree.

Ok, but you're importing the ``it's not proven'' argument into areas
that I never said it was used as one. Robert Watson has expressed a
definite interest in the IPI messaging code. Robert Watson also has tons
of work to do, writes several large emails per day and still somehow
finds time to actually get code done. I don't know how he does it all; I
certainly wouldn't have enough time in a day to do some things that he
does.

Additionally, the stability argument certainly wouldn't apply to
FreeBSD-native work, since that work is required for the advancement of
where FreeBSD wants to go. DragonFly's work in the same areas conflicts
with FreeBSD's on many points, which is why we have these two different
projects. For the code that can still be backported, please feel free to
search for developers that aren't pressed for time or money to port some
of these changes.

For me, it is certainly nothing more than a time constraint. I would
certainly love to port some of our features to FreeBSD, but I do not
have time. While you could argue that I certainly do have time for these
things, I do struggle quite a bit financially. Until I have a stable bit
of income with which I can support myself and still have some over, I'm
not going to be able to help with this. It just isn't practical for me.
While the situation may not be the same in the case of every developer,
I think you'll find it's difficult to find one who does have the time to
learn about and port these changes. I'm not sure what kind of scrutiny
and hostility you're referring to regarding any backported work. What
work has been backported that has been subject to this scrutiny (and
especially the hostility)?

The fact is that DragonFly exists to explore a different development
path. If you think that DragonFly exists to replace FreeBSD, I think
you're horribly mistaken. DragonFly is implementing its features in one
way; FreeBSD is implementing its features in another. 

The Perforce model is no different than how some of our developers work.
Joerg has his own patches playground, so does Simon and many other
developers. I've set up a repository that housed the PF 4 DF effort, as
well as Simon's slab allocator and several other projects. I do find the
work that Joerg does quite inaccessible because it is not accessible. If
I want to address anything that he's busy with, I need to directly
contact him about it and he's frequently reluctant to show the code. The
Perforce model has centralized this personal development and it makes it
easier for others to track the development of other developers who are
working in fields that they're also interested in. I would personally
really appreciate seeing our developers being encouraged to make use of
a central repository for their work. When I do have time for projects, I
find that it's spent in ``paperwork'': by the time I figure out who is
doing a project, I don't have time to look at it and contribute to it
any further. I don't personally think that's a good thing.

>     Personally speaking, I think we've proven our model in all aspects 
>     except performance.  What we are doing is clearly far more maintainable.
>     FreeBSD has a bit of a hidden beast problem in the maintainability 
>     department.  When an original author takes a vacation or stops working
>     on something, the maintainability problem hits them squarely in the face,
>     but its hidden as long as the original authors continue working on the
>     code.  That doesn't bode well for the future.  On the otherhand, there
>     have been half a dozen instances where people have come in cold and
>     done bug-free or mostly bug-free (meaning fixed in a day or two)
>     work on core pieces of the DragonFly code base.  Without any prior
>     instruction Jeffrey Hsu was able to thread and message most of the network
>     protocol stack using my LWKT messaging primitives.  Joerg was able to do
>     major cleanups of the namecache code with only one or two emails
>     between us.  David Xu sent me a TLS patch out of the cold that adds
>     code to the core LWKT switching assembly without any instruction.
>     Richard Nyberg debugged a namecache issue out of the cold.  I consider
>     these and other events as undeniably proving the maintainability of 
>     our codebase.

Jeff, Joerg and David are also quite apt developers. I don't know
Richard by name, but he doesn't look like someone I'd classify as a
beginner, either. I think all of these people understand how the FreeBSD
mutex model works and would be able to contribute to that field if they
were so inclined.

>     On the performance front... I absolutely refuse to rush into removing
>     the BGL.  I don't give a damn what excuses the FreeBSD people are making
>     with regards to unproven performance.  They are flapping their mouths
>     a lot about unproven this and unproven that, but they aren't actually
>     thinking theory and that is a serious mistake.  I want to get our
>     codebase using mostly MP clean algorithms BEFORE I start actually 
>     turning off the big giant lock.  So e.g. the threaded network protocol
>     stacks are using MP clean algorithms, but they aren't 100% MP clean
>     yet and so the BGL is still turned on.  We *KNOW* that those pieces
>     which are now MP clean, stable, and well tested, are not likely going
>     to be the cause of any bugs when we start to deal with the remainder
>     down the line.  That's what is important.  Not only that, but since
>     it is a threaded subsystem we have an ability to turn off the BGL on
>     a thread-by-thread basis, and even do it on the fly with sysctls,
>     which is a far saner development model then FreeBSD's 'oh lets throw
>     mutexes around all this junk, turn off the BGL, and pray' methodology.

I'm not suggesting that we do this. I didn't say that FreeBSD was
uninterested in importing stuff because it doesn't perform. I don't know
anybody who has.

What I _DO_ know is that I see plenty of claims in public places that
profess the speed of {Subsystem} in DragonFly versus FreeBSD. These
claims are not only unbased, they're not made in a diplomatic way (and
can come across quite offensively) and are on occasion outright false.

I'll restate: I am suggesting, for our benefit later, that we work on
developing some performance tests of certain features we believe to have
improved in speed.

Several people involved with the project claim now that we blow FreeBSD
out of the water (performance-wise) and this is where my comment comes
from. Since we eventually aim to be a high-performance OS, from what I
understand, I think it would be to our advantage to, at some point,
develop some benchmarks. That's all I'm saying there.

>     There is definitely some UP performance degredation from e.g. threading 
>     the network protocol stack, as one person's recent routing tests have
>     shown.  But considering the fact that we haven't actually tried to
>     *optimize* the messaging yet the numbers are pretty much in-line with
>     what I would expect.  More to the point, the *theory* behind getting good
>     performance out of a threaded subsystem is sound, primarily the 
>     ability to queue more then one piece of work before switching threads,
>     and I see such a clear path for optmization and improvement (without
>     having to resort to 'hacks') that I am confident that we will be able to
>     soundly thrash the mutex model when all is said and done.  Certainly,
>     no matter what, we will come close, and that would be a win too
>     considering the vast differences in maintainability between our code 
>     and FreeBSD's. 
> 
> 					-Matt
> 					Matthew Dillon 
> 					<dillon@xxxxxxxxxxxxx>

Good code is a good first step.

It's really not a contest, nor is it a war. If it is, I'd like someone
to please tell me so I can get out of it.

Thanks,

Devon