Installing Mailing Lists using EoC and MHonArc
Note: this documentation is fairly old, and may be out-of-date. I no longer use EoC or MHonArc, having moved my mailing lists to SourceForge.
Background
I went through a lot of pain trying to get mailing lists up and running on daystrom. I originally planned to use Mailman, since that has been what I'm using at work. Initially, I tried to get a backport of the latest version (2.1.4) working, since earlier versions are reported to have some problems. That failed. Then, I tried the version in 'woody'. That one absolutely won't work unless the webserver is running as group www-data. If you are using a different group (as I am) you have to rebuild the package. The heck with that.
I dug through a lot of other mailing list managers and archivers, and eventually ended up with Enemies of Carlotta (EoC) in combination with MHonArc for HTML archiving. EoC is small, clean and easy to use. The Debian package doesn't even come with extraneous stuff like crontabs or anything. It just knows how to manage a list.
These two together arguably offer a better solution than Mailman in conjunction with Pipermail. The result is easy to backup, and MHonArc does a better job with threads than Pipermail, which I've never really liked anyway.
The whole thing was a bit of a pain (basically a whole afternoon/evening's worth of effort), but in the end I got what I wanted and I don't think it was horribly difficult. However, it was a bit more complicated than I had hoped for, and I need to document what I did here so I don't forget anything.
Packages
First, there is no EoC package for Debian 'woody', only for 'sarge'. However, the package is so simple - it depends only on Python 2.1 - that I was able to build it right on daystrom. So, I'm running unstable's package out of my own custom-built .deb. Maybe I'll pin it to unstable later or something, I don't know. I probably should have just downloaded the .deb from packages.debian.org, but I wasn't thinking clearly.
There is a version of MHonArc for 'woody', but it turns out the latest version is already backported (in the same backport site as Spamassassin). So, I also ended up with unstable's version of this package:
/root# apt-get install mhonarc
Reading Package Lists... Done
Building Dependency Tree... Done
The following NEW packages will be installed:
mhonarc
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 1863kB of archives. After unpacking 8192kB will be used.
Get:1 http://people.debian.org stable/main mhonarc 2.6.8-2 [1863kB]
Fetched 1863kB in 8s (231kB/s)
Selecting previously deselected package mhonarc.
(Reading database ... 27887 files and directories currently installed.)
Unpacking mhonarc (from .../mhonarc_2.6.8-2_all.deb) ...
Setting up mhonarc (2.6.8-2) ...
Scripts and Cron Jobs
Originally, I was going to use the standard Debian list user to deal with mailing lists. However, I decided that I wanted to stick with my "public services" model, and I figured I'd be better with my own user if I wanted to do things that way. In order to make this work, I had to create that new user, two scripts, a logrotate conf file and three cron entries, plus new Exim configuration.
The whole thing runs as the listmgr user, including the interface to Exim. The listmgr user was created with primary group public, and owns the new /opt/public/lists directory:
/opt/public/lists> ls -la total 48 drwxrwsr-x 6 listmgr public 4096 Feb 9 22:21 ./ drwxr-sr-x 10 public public 4096 Feb 9 16:42 ../ drwxr-s--x 3 listmgr public 4096 Feb 9 21:40 .enemies-of-carlotta/ -rw------- 1 listmgr public 842 Feb 9 20:52 .logrotate.conf -rw------- 1 listmgr public 962 Feb 9 16:46 .profile lrwxrwxrwx 1 listmgr public 20 Feb 9 16:53 eoc -> .enemies-of-carlotta/ drwxr-sr-x 3 listmgr public 4096 Feb 9 21:40 listarchives/ drwxr-sr-x 2 listmgr public 4096 Feb 9 21:40 util/
Mailing list information is stored in the eoc directory (I made the eoc soft-link because I think the .enemies-of-carlotta file name is a real pain in the ass). HTML archives are stored in the listarchives directory. See the sections below for more details on these directories.
EoC creates a fairly large log file containing all of the mailing list traffic. This file is apparently not rotated or truncated. So, it seems best to have logrotate deal with it. This is ~listmgr/.logrotate.conf:
/opt/public/lists/eoc/logfile.txt {
missingok
weekly
compress
create 640 listmgr public
rotate 5
}I also ended up with these three cron entries that take care of periodic maintainence for the listmgr user. This is listmgr's crontab:
# Just after midnite, run the logrotate script on list-related entries
3 0 * * * /usr/sbin/logrotate -s ${HOME}/.logrotate.status ${HOME}/.logrotate.conf
# Every 15 minutes, update the mailing list archive
0,15,30,45 * * * * ${HOME}/util/listupdate
# Sunday at 3:00am, cleanup Enemies of Carlotta
0 3 * * 7 /usr/bin/enemies-of-carlotta --cleaning-womanThe listupdate script is just a quick script I pulled together to run the MHonArc incremental-update command discussed below.
Integration with Exim
Basically no mailing list managers integrate automatically into your box's MTA, since every MTA has a different way of supporting lists, aliases, etc., etc. Exim usually requires hand-editing of alias files, etc. This initially had me annoyed, but then I found this thread at the EoC site http://liw.iki.fi/lists/eoc@liw.iki.fi/msg00019.html.
This thread describes how to "hook" EoC into Exim3's configuration. Essentially, you force Exim to look for a valid list among the config files in EoC's working directory, and if one isn't there, you fall through to the other normal lists.
There are basically three pieces to the Exim changes, all of which take place within /etc/exim/exim.conf: constants, a new transport and a new director.
I put the constants at the bottom of the "MAIN CONFIGURATION SETTINGS" section:
EOC_HOME = /opt/public/lists EOC_COMMAND = /usr/bin/enemies-of-carlotta EOC_CONFIG = EOC_HOME/eoc EOC_USER = listmgr EOC_GROUP = public EOC_DOMAINS = mydomain.com
Then, I put the EoC transport at the bottom of the transports section:
eoc_transport:
driver = pipe
command = EOC_COMMAND --incoming --quiet
current_directory = EOC_HOME
home_directory = EOC_HOME
user = EOC_USER
group = EOC_GROUP
environment = "RECIPIENT=${lc:$local_part}${lc:$local_part_suffix}@$domain"The --quiet switch is required because EoC is fairly verbose and Exim will hang if there's too much output from the command.
Finally, I put the EoC director at the top of the director's section:
eoc_list_director:
domains = EOC_DOMAINS
driver = smartuser
suffix = -help : -list : -owner : -setlist : -setlistsilently :
-ignore : -subscribe* : -unsubscribe* : -subyes* : -subapprove* :
-subreject* : -unsubyes* : -bounce* : -probe* : -approve* : -reject* :
-setlistyes* : -setlistsilentyes
suffix_optional = true
require_files = +EOC_CONFIG/${lc:$local_part}@$domain/config
transport = eoc_transport(The suffix= entry should all be on one line, but I can't display it like that here.)
It is very important that the director be at the top so that errors fall through to the other available directors, otherwise every "user not found" error will look like an EoC error instead.
The require_files line in the director caused me some consternation. Basically this line tells Exim, "If this file exists, use this director. Otherwise, go on to whatever directors are left below this one." At least, that's what's supposed to happen.
However, it only works if Exim's effective user and group (mail:daemon on a Debian system) can read the directories that are part of the path. The require_file field actually allows a user/group value which specifies the user and group to use when looking for the required file. However, that can't work because by the time it gets to this point, Exim has already dropped privileges and can't switch to that user/group http://www.exim.org/pipermail/exim-users/Week-of-Mon-20000103/015959.html.
The solution I chose was to add o+x permissions on the directory, because I want it to be owned listmgr:public.
Also, note the + at the beginning of the require_files value. This tells Exim to treat "access denied" as failure to find the file. If the + isn't there, then Exim treats "access denied" as a temporary failure and defers delivery of the message. This creates havoc when there are permanent permissions problems (as described above) that prevent Exim from reading the EoC config directories at all.
Creating lists
Creating a list with EoC is fairly simple. Just log on as the user you want to own the list (in this case, listmgr) and do something like this:
alias eoc='enemies-of-carlotta' eoc --name=mylist@mydomain.com --owner=owner@mydomain.com --create eoc --name=mylist@mydomain.com --subscribe owner@mydomain.com
There are a lot of other options, but this is as easy as it gets. If you've already done the Exim configuration (above) then the list will start working immediately.
The list itself will be created in the user's ~/.enemies-of-carlotta directory, along with a number of other files:
/opt/public/lists/.enemies-of-carlotta> ls -l total 16 drwx--s--x 6 listmgr public 4096 Feb 9 21:40 mylist@mydomain.com/ -rw-r--r-- 1 listmgr public 4393 Feb 9 21:41 logfile.txt -rw-r--r-- 1 listmgr public 32 Feb 9 21:40 secret /opt/public/lists/.enemies-of-carlotta/mylist@mydomain.com> ls -l total 24 drwx--S--- 2 listmgr public 4096 Feb 9 21:41 archive/ drwx--S--- 2 listmgr public 4096 Feb 9 21:40 bounce-box/ -rw-r--r-- 1 listmgr public 174 Feb 9 21:40 config drwx--S--- 2 listmgr public 4096 Feb 9 21:40 moderation-box/ -rw-r--r-- 1 listmgr public 60 Feb 9 21:40 subscribers drwx--S--- 2 listmgr public 4096 Feb 9 21:40 subscription-box/
The logfile.txt file is a running log of activity. It should definitely be rotated with some frequency. Also, the manpage suggests running EoC with its --cleaning-woman option periodically to deal with bouncing addresses and other cleanup. Both of these items are handled by cron entries (see above).
You'll likely want to modify the configuration file for the list, to make it work as expected. This is what I use:
[list] language = subscription = free posting = auto archived = yes mail-on-forced-unsubscribe = yes mail-on-subscription-changes = yes owners = owner@mydomain.com
Basically, I say: anyone can subscribe, only subscribers can post, archive the messages, and send emails when things change.
If a non-subscriber attempts to post, then I will get a moderation email and I'll have to accept or deny it. For the time being, I'll manage this by hand, but if I start getting a lot of spam, I may just script something to throw away any message that requires moderation.
In the meantime, I have had a problem with this, and I wrote a new mode for EOC called "rude". See http://liw.iki.fi/lists/eoc@liw.iki.fi/msg00361.html.
Archiving HTML
Just like EoC, MHonArc does one thing and does it well. There are no extra frills, no crontabs, etc. with the Debian package. Everything has to be done by hand. However, the good news is that it's fairly easy.
MHonArc operates on either maildir directories or on standard mailbox files. The simplest way to use it is to do something like:
mkdir archive && mhonarc -quiet -outdir=archive mailbox
This will regenerate an entire archive based on a mailbox. You can also use MHonArc in "incremental" mode:
mhonarc -quiet -add -outdir=archive mailbox
The -add option tells MHonArc to only add new messages into the archive.
To use MHonArc on a maildir rather than a mailbox, you have to list each file individually, i.e.
mhonarc -quiet -add -outdir=archive maildir/*
MHonArc actually does a pretty good job of inferring threads, etc., just like mutt. There are actually two different indices, a date-based (chronological) index and a thread-based index. I prefer the thread-based index, but it's worth having both around:
/opt/public/lists/listarchives> ls -l total 4 drwxr-sr-x 2 listmgr public 4096 Feb 9 22:15 mylist/ /opt/public/lists/listarchives/mylist> ls -l total 20 -rw-r--r-- 1 listmgr public 770 Feb 9 21:45 maillist.html -rw-r--r-- 1 listmgr public 3065 Feb 9 21:45 msg00000.html -rw-r--r-- 1 listmgr public 5067 Feb 9 21:45 msg00001.html -rw-r--r-- 1 listmgr public 657 Feb 9 21:45 threads.html
To display the list archives on the Cedar Solutions website, I added an alias:
Alias /listarchives /opt/public/lists/listarchives
and then also created a decent index.html (residing in /opt/public/http) and soft-linked it into /opt/public/lists/listarchives.
There doesn't seem to be a great way to hook MHonArc into the EoC delivery process to have it dynamically update the archive every time a message is sent. (I guess I mean, there probably is a way, but I haven't found it.) I'm not sure that this matters. I think that a better option is just to run a cron job once every so many minutes to add the new messages. There'll be some delay, but it won't be horrible, especially with the volumes that I'm going to have. That's what the listupdate script (discussed above) is for.
MHonArc supports a fairly compex templating system that can be used to change the look-and-feel of the pages in the archive, i.e. add a header, add a footer, etc., etc. This might be nice someday, but right now, I don't particularly care about it. Instead, I just used the -title and -ttitle options to set the titles on the threads.html and maillist.html pages. That's good enough for now.