Re: Preliminary restructuring layout (was Re: sys/ tree re-structuring proposal)

[Matthew Dillon <dillon@xxxxxxxxxxxxxxxxxxxx>]

     Well, excellent, it looks like I didn't mess things up too badly :-)

     I'm going to be working on the module build today but I thought I would
     also explain some of things that were done including some that might
     look a little wierd.

Additional cleanups everyone should do to reduce confusion:

    # remove random non-archive garbage from your sys/ tree and then
    # cvs update in order to allow cvs to remove dead directories.
    # (using the suggested 'update -Pd' line in your ~/.cvsrc)
    #
    cd /usr/src/sys
    find . -name '*.bak' -delete
    find . -name '*.orig' -delete
    cvs update


Obvious changes:

    netproto/{atalk,atm,ipsec,ipx,key,natm,ncp,ns,smb}

	Generally replaces top level directories netatalk, netatm, etc.  This
	change went fairly smoothly and I think it is an improvement.

    net/{accf_data,accf_http,bridge,disc,dummynet,ef,faith,gif,gre,i4b,ip6fw,
	 ip_mroute,ipfilter,ipfw,ppp,ppp_layer,sl,sppp,stf,tap,tun,vlan}

	Generally move 'internal' network interfaces and partitionable
	infrastructure to subdirectories.

	net/ipfw and net/ip6fw are experiments.  In particular, ip6fw is
	experimentally moved from netinet6.  I haven't moved other things
	from netinet6 because I really don't know how good an idea it is
	yet.

	i4b was the most complex thing to clean up, particularly note that
	the architecture-specific portions of i4b were moved from i386/include
	to net/i4b/include/i386.  This was done as an experiment to see how
	well things could be organizde with architecture-specific pieces broken
	out and associated with their subsystem rather then jumbled together
	in one directory.  It required creating an 'i4b_machine' directory
	in /usr/include in order for programs to access machine-dependant
	portions of i4b (since <machine/BLAH> no longer has those files).

    dev/* -> dev/{agp,atm,crypto,disk,drm,hfa,misc,netif,powermng,raid,serial,
		sound,streams,usbmisc,video}/*

	Moved the device drivers from the jumbled mess under dev/ to sub
	categories.

    *fs,miscfs/* -> vfs/{coda,deadfs,fdesc,fifofs,gnu,hpfs,isofs,mfs,msdosfs,
			nfs,ntfs,nullfs,nwfs,portal,procfs,smbfs,specfs,
			ufs,umapfs,union}

	All the VFS subsystems have been moved to the vfs/ directory.  mfs is
	no longer under ufs and ufs and ffs have been combined together and
	is just named ufs.

	I really liked how this came out.

    emulation/{ibcs2,linux,posix4,svr4}

	I moved the random top level emulations and ibcs2 (which was under
	i386) into emulation.  It may have been a mistake to move posix4 to
	emulation and I might move that back to the top level.

	Eventually emulation/ will be converted to a userland middle layer
	as previously discussed, so it was important to partition it into
	its own subdirectory.

	Architecture specific files were moved under their appropriate 
	emulation subdirectory.  e.g. instead of i386/linux we have
	emulation/linux/i386 for linux-specific i386-specific files.

    bus/{cam,canbus,eisa,firewire,iicbus,isa,mca,pccard,pci,ppbus,smbus,usb}

	All the bus architectures were separated out and moved into their
	own subdirectory.

	I may have made a mistake moving cam to bus/cam.  It's hard to say 
	what 'cam' is but it seems to be referenced in enough disparate 
	places that leaving it at the top level is probably the better choice.

    modules/* -> associated directories

	All the module Makefile's have been moved to their associated
	subsystems.  The 'modules' directory hierarchy has been deleted.

	The Makefile's are currently named 'Makefile.module' just so I know
	what I have to go in and fix.  They are not part of the build system
	yet.

    netgraph/*.h -> netgraph/<subsystem>/*.h

	When I was reorganizing netgraph I had to reincorporate the
	modules/ Makefiles into it which is why I moved the subsystems
	into their own subdirectories.  I actually didn't mind all the
	subsystems being at the toplevel (in netgraph/*) but I didn't see
	an easy way to incorporate the modules Makefile's that way.

	This turned out to be a real hassle for /usr/include file generation
	since there are a lot of netgraph related userland utilities.  It
	might make sense to undo this part of the move but for now I am 
	going to concentrate on making the modules build work.

Inobvious changes:

    By removing the use of -I- I changed many of the absolute #include file
    paths that were being used to reference header files in the same 
    directory as the source file into relative #include file paths using
    quotes, e.g. "blah.h" instead of <a/b/c/blah.h>.  This generally made
    it a whole lot easier to move subsystems around and I also believe it
    is a good idea just on principle.  This also makes inter-subsystem
    dependancies far more apparent in the codebase.  I did not universally
    'fix' all the #include's for locality of reference because it is an
    aweful lot of work, but I made a good dent.

    However, 'config' generates header files for devices named <DEVNAME>.h
    in the compile/ directory, which conflicts with similarly named header
    files in the source tree.  Originally the source code used "blah.h"
    to get at the compile/ directory header file and <a/b/c/d/blah.h> to
    get at the source-version.

    To solve this I changed 'config' to generate use_<DEVNAME>.h instead of
    <DEVNAME>.h.  I think this change will be well received because it really
    makes things clear in the source modules and it fits well with the
    opt_<SUBSYSTEMNAME>.h files that config already generations for options.

    /usr/include generation required considerable work.  Anyone making further
    changes to /usr/src/include/Makefile should run them by me before
    comitting as there are some gotchas (which I documented in the Makefile).
    I added a lot of 'compatibility' softlinks to reduce the amount of work
    that had to be done in /usr/src.  Please examine your newly generated
    /usr/include to see the new organization.

    UFS used to be broken up into ufs/ufs, ufs/ffs, and there was also ufs/mfs.
    Now we just have vfs/mfs and vfs/ufs, with all the files that were in
    ufs/ufs and ufs/ffs now in vfs/ufs.  There were no conflicts in making
    this change and its about time that we stopped abstracting ffs as a layer
    on top of ufs, at least in regards to source file organization.

Final Note:

    None of the changes are set in stone and I am sure we will be doing 
    additional reorganization work, especially in net/, netinet/, and
    netinet6/.  However, it isn't easy to reorganize files due to the huge
    number of #include dependancies so I am going to let things settle for
    at least a month, maybe longer, before any additional major rewiring.

						-Matt