Re: first stab at simple mailer

[Matthew Dillon <dillon@xxxxxxxxxxxxxxxxxxxx>]

:The queue works like this:
:
:- create a temp file
:- this queue file has a "header" prepended
:- read the mail from stdin and write to the temp file
:- each recipient gets assigned a queue id, which is the filename at the s=
:ame time
:- the temp file gets linked to the queue ids
:- when a recipient was processed, the link gets removed
:
:this way I only have one instance of the mail on disk and I don't need ad=
:ditional files to track for one mail.

    It's a fair idea, but very very filesystem intensive.  Using softlinks
    might be less invasive (at least then the inodes don't have to be
    updated every time you make a delivery).

    An even better solution would be to generate a worklist in one
    or more files... maybe chunk it up, 20 recipients per file or something
    like that.  Doing it that way then gives you the option of pre-sorting
    the worklist when generating it (in the future) without having to also
    change the backend code that processes it.

    You need to do this anyway to avoid making a separate connection to
    any external entity for each recipient in a list.  It's not optional.

:>     * Remote delivery, which you have as 'XXX implement me'...  you nee=
:d
:>       to be able to sort by the target domain's MX lookup and you need =
:to
:>       do MX backoff & forwarding for any remote delivery.  It isn't too=
:
:>       bad but DNS lookups can be difficult to implement efficiently.
:
:I don't really care about efficiency.  This mailer is supposed to be a lo=
:w-end thing, just able to enable a base system to deliver common day mail=
: loads (by periodic, cron and users).  This is *NOT* supposed to be a hig=
:h-performance mail server.  Anybody who needs any sort of sophisticated m=
:ail processing can use the mailer they want.

    Yes, but I think it still needs to be designed in such a way that
    the efficiency is reasonable, and to leave the door open for 
    future work to make it even more efficient.  Continue reading on
    for more on this.
    
:>     * Configuration file. Yup. you need one :-).
:
:Not sure what I want to configure.  Maybe a smart/relayhost, but nothing =
:more.

    The various hardwired constants you have at the moment at the very
    least.  Since it will be interfacing with the user you must also be
    able to configure maximum mail header and body sizes.
    
:>     How serious are you about this?  There are a number of issues that
:>     would have to be addressed, even for a simple mailer.  Handoffs for=
:
:>     .forward files (including processing exit codes properly),
:
:What do you mean with handoff?  I want to support .forward, but only for =
:file and address forwards, not for command forwards.

    You have to support command forwards, or it isn't useful.  Many, many
    people use things like procmail and it must work in the base system.

    It actually is not all that hard to do, you basically just create a
    pipe and fork/exec (with appropriate permissions and ownership changes)
    the program, pipe the data to it, process the response (if any), and
    handle the exit code.

:>     supporting
:>     /etc/mail/aliases and /etc/mail/virtusertable,
:
:yes, aliases is in work.  I've never heard about virtusertable before, an=
:d I don't see where this might be necessary in a simple mail setup.  This=
: mailer accepts only local mails.  No listening on port 25.  As soon as y=
:ou need mail rewriting, a "big" mailer should be installed.

    You need it even for simple mail setups.  Hosting multiple domains is
    something that is very common these days and it has to be supported
    out of the box.  But, more importantly, the local mailer either needs
    to be told that a recipient is a local recipient, or it needs to be
    able to figure out in absolute terms whether the recipient is local
    or not, or you risk creating severe mail loops.

    With regards to these files... don't go the DBM route that sendmail
    took.  Just index them internally (in memory) and do a stat() check
    to determine when they have changed (so the daemon doesn't have to be
    restarted).

:>     and handling a more
:>     sophisticated local delivery check (using a DNS lookup on the targe=
:t=20
:>     domain to see if its a CNAME to the actual host, or something simil=
:ar)....
:
:I thought about this, but I decided to go down the easy road for now.  Th=
:is shouldn't be much of an issue.  Again, the mailer is not designed to a=
:ct as an MX.  How do I find out if an IP address is local?  This check wo=
:uld basically be the only important check.

    I don't think it is optional.  The mailer really needs to be able to
    determine whether a recipient is local or not.  You could list the
    local domains in a file (in the case of sendmail, the file is called
    /etc/mail/local-host-names) and not bother with the DNS lookup.

    Doing a DNS lookup is far more convenient, though (in addition to having
    the file), because then the user doesn't have to maintain a list of
    local hostnames.

:>     There are many other issues as well, including mail size limitation=
:s,
:>     smarter queueing mechanisms (sorting by target recipient to prevent=
: a
:>     stalled target host from stalling the entire queue), handling
:>     out-of-disk-space issues gracefully, etc.
:
:There is no global queue manager.  Each mail gets processed on its own.  =
:Yes, I am aware of the fact that this might open two connections to a des=
:tination mailer for two mails.  However, this is a simple mailer and for =
:simple mail loads, this won't be a problem.

    You could make it an MX-forward-only mailer, sure.  That's reasonable.

    But you still need to efficiently queue the mail for forwarding.  You
    can't just make separate connections to the target for each recipient.
    That makes it impossible for the target to optimize its own mail storage.
    In a MX-forward-only setup you still basically want to make just a
    single connection to the forwarding host and pass all the non-local
    recipients to it for any given piece of mail.

    In a direct-delivery setup you still need to optimize multiple recipients
    going to the same target.  Again, its an absolute requirement for even
    a simple mail system.  It's unfair to burden target systems with separate
    connections that make it impossible for those systems to efficiently
    store the mail for further processing (just as you want to efficiently
    store multiple recipients in your own code, so must other targets be
    able to do the same thing with the mail you send to them).

:Out of disk space is being handled:  if the mail can be queued, everythin=
:g is fine.  If not, the mailer errors out (nothing else it can do).
:
:I don't think I want to care about mail size limitations.  That's the job=
: of a "real" mail server.

    It isn't something that has to be implemented right off the bat, but
    generally speaking you need to be able to set limits for anything
    that interfaces to userland.  You don't want root mail from cron jobs
    to break because some userland program filled up the mail queue
    (intentionally or not).

:...
:
:>     the back-end over a localhost socket and pass creds with sendmsg() =
:
:>     and SCM_CREDS (with regards to a mail.local equivallent).
:
:I don't really want to split this into backend daemon and frontend tool. =
: Besides, where is the difference?  One of them always needs to run with =
:elevated privileges.  I do *not* want to make it setgid wheel.  Just setg=
:id mail, where mail may write to /var/spool/mqueue and to /var/mail/*.  I=
:t's really supposed to be simple.  No running daemons required.
:
:cheers
:  simon

    The difference is that the user can manipulate the frontend tool far
    more then the user can manipulate the backend tool.  For exmaple, ^C,
    or ^Z, and so on and so forth.  It's very dangerous.  Having SUID or
    SGID programs directly manipulate subsystems is a bad idea in general.

					-Matt
					Matthew Dillon 
					<dillon@backplane.com>