a proposal for the packaging system

[Raphael Marmier <raphael@xxxxxxxxxxx>]

Sorry, this is long, I appreciate your patience.

I have been following the progress of DragonFlyBSD since the beginning and I 
must tell one of the things that get me most excited about it is the prospect 
of real innovations regarding package management.

However, the current discussion on the matter seems to go nowhere everytime, as 
you pointed out. This is not suprising, as a packaging systems finds itself 
naturally trying to satisfy everybody while having to stay simple. So guys, 
everybody is right, and that's the problem.

If we want to get things a bit clearer, we must deconstruct the problem. A 
packaging system as a whole is roughly made of the following parts:

1) The way packages are instaled on the filesytem. (location, pkgdb, ...)
2) The basic tools responsible for taking a package and installing it in the fs.
3) The package format
4) The port/package collection its structure, organisation and content.
5) A build system, package generating system, tools for fetching distfiles, 
misc. ancillary tools, etc...
6) Package management tools providing high level functionalities based on the 
"primitive" provided by point 2 and 4.

The reason it ends in a flame war everytime is that everyone talks about a 
different part.

Now, serious work should be to isolate the pros and cons of each solution at 
each level. Ex.: Debian may be better in the packaging format (dpkg) and 
obviously fbsd ports is better with its build system (build time options).

For every improvement proposed, it has to be shown what level would be affected 
and the consequences weighted on cost/benefit. Maybe someone should start a 
wiki on the subject and people would submit their proposal they already exposed 
in the list in this formalised way. Then, it'll be easier to take decisions.

Hastilly ditching freebsd's port collection (acting at level 4) would be 
kamikaze, IMHO. We can still act at other levels, especially level 1 and 2.

In the meantime, I reread Matt's memo on packaging and obviously the bitterest 
grief with current solutions is the inability to do something like install an 
new version of libncurse with some funny option without breaking the 
application that relie on a "normal" install of it. Now, that would be some 
_innovation_, and that would definitely differenciate DBSD from any other unix.

Now the debate is wether this is achievable only through "VFS Voodoo" (level 
1), or there is a way to do that with a new wizardry of the port system (level 
3,4,5).

IMHO, the second option is bound to fail like all others before, because it 
amounts to try to please every user and be suitable for every use. We know this 
is not achievable. The debian ports are the closest to "success" in this 
approach, yet still fail. And starting from scratch is costly.

The VFS Voodoo approach needs just dealing with level 1 and 2, which are 
entirely in the os and not in the port collection, in a way that would be 
transparent not only to the port collection, but to the higher level tools such 
as portupgrade as well.

Matt's Filesystem Overslay proposal looks daunting, but it can be implemented 
in a simple, generic, and I hope, elegant way, which I'm going to try to 
describe now.

Filesystem Overlay for DragonflyBSD:

A special db store lists of filesystem overlay. Each list describes a single 
overlay: the list names the directory or file which it applies to and a list of 
replacement directory or file (must be same type). Each list has a unique 
identifier, which should allow alphanumerical name.

When the executable file is runned, its environment is set up to indicate the 
overlay that applies to it (read-only var) When it tries to read the overlaid 
dir (let's say /usr/local/lib), the system notice the overlay, fetch the proper 
list, and tries each listed dir entry in turn (order matters), and stop at the 
_first_ matching one. If none match, then the actual /usr/local/lib is tried 
last.

So when you install a new version of ncurse that is not supported by your, say 
midnight commander, what happens?

The installation tool (level 2) notice the already installed version of the 
same port, finds in the pkgdb that midnight commander needs it. So it move the 
installed package to a new directory /usr/local/prev/libncurse-1.5-xxx/... 
where xxx is a local version number derived from the date, updates the pkgdb 
entry of the old libncurse to match the new position, updates its name to 
include the new local version number, updates the other pkgdb entries to point 
to this new package name, and create a overlay list applying to /usr/local/lib 
pointing to /usr/local/prev/libncurse-1.5-xxx/, and finally tags the midnight 
comander executable(s) with this list.

At this stage, room has been made for the new package which can be installed 
normaly.

A cron job can scan through the fs and the special db to remove dangling 
entries, etc... when packages have been removed.

The nice thing is that system is not limited to ports and can be used even for 
surprising goals, like managing access allowed to an executable. It suffice to 
put an overlay on, say, /etc/ refering listing /var/overlays/myprog/ and put a 
passwd file in that latter dir with perm at 700 and this program only cannot 
read /etc/passwd, because the system hit /var/overlays/myprog/passwd first and 
uses its authorisations.

As a conclusion, this system ends up _creating packages_ (new versions tag) out 
of known working combinations of old-new ports. This ends up being much more 
natural than to trying to enforce it in the ports. Its a bit like capitalism 
versus planned economy, if I dare.

Raphael Marmier