Re: Packaging (Re: Goals for first release (June/USENIX))

[Matthew Dillon <dillon@xxxxxxxxxxxxxxxxxxxx>]

    My own ideas on an installation framework have not yet solidified, but
    the more I think about things like variant symlinks and VFS environments,
    the more it seems to me that the actual physical install location should
    be in a single, properly named and versioned subdirectory, e.g. like
    /opt/<pkgnameandrev>, or /usr/local/<pkgnameandrev>, or something like 
    that.

:...
:haven't been updated), go through the usual configure/make steps but
:type some variant command of "make install", and have the whole thing
:not only installed exactly as usual, but registered in the package
:database with a proper +CONTENTS file so that it can be uninstalled
:cleanly if desired.

    I would like to be able to do this too, which is how I cam up with
    the VFS environment idea.  The trick is to not burden the developer,
    so if we were to do all of the above within a VFS environment then,
    in fact, we could also 'make install' within that environment and
    then tell the real installer which directories represent the installed
    files.

    The real installer would pull them out of the environment and put
    everything in /opt/<pkgnameandrev>/, so no 'CONTENTS' file would 
    actually be necessary, and would also create an environment file,
    say /opt/<pkgnameandrev>/ENVIRONMENT, that tells the system how to wire
    the package so we see it where we expect to see it.

    In this example, the VFS environment itself is just a temporary affair
    used to during the build and installation phase and would not have to be
    used for the run-time phase (at least not for most packages).

    The advantage of this is that there is a very clearly defined place to
    look for (A) the installed package's raw files, and (B) the information
    required to wire or unwire those files into the system (the ENVIRONMENT
    file).

    Now if you think about this, it leads to all sorts of advantages:

    * Human readable debugging and investigation.  Sysops know exactly where
      to look to understand where a package is installed and sysops know
      exactly where t look to understand how a package is wired.

    * Utilities.  e.g.  'whoowns /usr/local/bin/soffice' would return
      something like:

      /usr/local/bin/soffice	varsym for soffice-1.1, soffice-1.2, and
				soffice-1.3, set SOFFICE_VERSION to select.

      The idea being that you could identify any file in the system this 
      way.

    * Database rebuilding utilities.  Everything would be sourced in
      /opt/<packagename>/ENVIRONMENT.  If our package management 
      subsystem uses a database, the database would be easily reconstructable
      simply by scanning all the ENVIRONMENT files in /opt/*.

    * Resynchronization and Validation.  It is often the case where a system
      is partially copied onto a new HD, or partally destroyed through
      various means.  This methodology would allow us to 'rebuild' the
      installed varsyms from what we see in /opt, as well as get rid of
      those that are no longer attached to anything, perhaps by consulting
      an off-site database that indexes the ENVIRONMENTs for all
      available packages.

    * Conflict management.  We would use varsyms as much as possible to
      deal with conflicts.  Even more to the point, it would be posible to
      have a two-level system where for those packages which are severely
      manged between released (and e.g. install to totally different places),
      the primary varsyms in e.g. /usr/local/etc and a/usr/local/bin, would
      simply point to secondary softlinks within /opt/<packagename> itself.

      In those hopefully few cases where varsyms are not sufficient resolve
      conflicts, or paricularly badly written third party software (nah,
      nobody would ever do that!  Right!), then the actual run-time environment
      could end up being a VFS environment too.

    * Transparency.  Since everything looks 'normally' installed from the
      point of view of system management, we would easily be able to switch
      between base-system packages, which are installed directly, and
      package-managed packages, which are managed via /opt.  This would allow
      us to slowly migrate base-system packages to 'real' packages, at least
      for the more complex base-system packages.

    I just don't know if I can get to this before the first release, at least
    not on my own.  It's not just a matter of a chicken-and-egg issue, it's 
    that I think it is important not to leave too many loose ends in the
    system.  I had already made significant progress on asynch syscalls,
    upcalls, IPC, and other subsystems.  I need to finish those before I
    embark on yet another new subsystems.

    But I am not 'reserving' the package management subsystem either.  I
    think it would be an excellent project for a group of developers to
    undertake.  In fact, you don't even need to have working VFS environments
    to start building it... a chroot plus unionfs mounts would be
    plenty sufficient.  And if unionfs is still broken, that is certainly
    something I can help fix.  I would definitely like to have a hand in
    developing the ENVIRONMENT specification and in mapping out how the
    package manager would work, since I do have my own requirements (e.g.
    to accomplish the above) that I feel are necessary to create a compact
    but easily managed packaging system.  What I don't have time to do right
    now is the actual major programming.

						-Matt