Moving Whiteoaks.com — Building a mail server

Don’t do this. This is stupid. Running a mail server is a pain in the ass.

If your startup or family or organization needs a domain and email, just point your domain at Google and use their great mail, calendar, and shared documents tools. Avoid all the battles with spammers, black lists, hackers and security updates.

So why would I do this? Maybe because I always have — I’ve had a server connected to the net since 1995. I enjoy being master of my domain(s). In a perverse way it’s a challenging hobby as well.

My server for whiteoaks.com lives in the linen closet at the top of my staircase. It really doesn’t get enough cooling, and when the processor gets too busy it complains about the heat. The version of Linux on it is old and long unsupported. I’ve been having to build package upgrades from source — like gcc, apache, php (4 and 5), and mysql.

I know I’ve needed to build a new server, but I’ve been dreading putting together a machine and painstakingly migrating my services, users and domains to a new environment.

Meanwhile Jane and I have discovered the joys of streaming Netflix and Amazon to the home theater. That means sharing my server’s DSL line to download TV and movies — not great for either service.

I pay Verizon about $95 per month for a commercial DSL account with 3 mbps download and 768 kbps upload. For a server, 768 kbps is pretty slow. Moving to the cloud would give my server cloud-sized speed as well, and let me switch to a consumer internet account at home — cheaper and probably faster.

So it’s really long past time that I moved out of my linen closet and into the cloud.

Much techie stuff follows. Feel free to skip down!

I spent several hours pricing out servers from Rackspace and others, trying to figure out what it would cost me. Right now I run everything — email, DNS, a dozen or so web sites — from one eight-year-old server with 1GB of ram and 40GB of disk.

I wanted to upgrade my mail service to use Zimbra, and I knew it would require its own dedicated server with 1GB of ram. On Rackspace, that’s “flavor 3” and would cost me about $40 per month.

Here’s why I wanted Zimbra: It’s available in a free open source community edition. It’s built on top of all the open source packages I was already using and familiar with: Postfix, Clamd, Spamassassin, MySQL. Zimbra provides a first-class web mail client that would enable my users (mostly family) to do things like change their password and set up filters to route mail into folders.

Over my vacation, I did an experiment. (I did a lot of this using wi-fi on a Delta flight. A first-class seat makes for a delightful work environment!)

I spun up a 1GB Rackspace server using Ubuntu 10.04LTS as the base. I set up a subdomain “rs.whiteoaks.com” using Rackspace’s excellent DNS support. I made an MX record to point to the new machine mail.rs.whiteoaks.com. Rackspace DNS support even let me set the reverse DNS to resolve mail.rs.whiteoaks.com to this server — essential on an email server!

(Rackspace DNS is so good that I can easily drop my own BIND DNS server, with no regrets. Let them deal with DNS security!)

On the new server I went through the Zimbra install script, interrupted only to install a couple of missing packages, and found myself a few minutes later with a genuine working email server that would send and receive email.

As an experiment, I made an image of that server, destroyed the server, and created one from the image. It came up okay, but I couldn’t connect IMAP clients to it, such as my Android phone email. (I later realized that the self-signed SSL certificate created during the Zimbra install needed to be recreated for the new IP address.)

Zimbra does not include a first-class mailing list manager. I host mailing lists for several astronomy clubs, OTASTRO, Bridgemojo and such — not huge lists, but as large as a few hundred members.

The best non-commercial mailing list manager for many years has been Mailman, supported by the community at http://list.org. I’ve always used
Mailman for my lists.

This led to much teeth-gnashing as I started researching how to integrate Zimbra and Mailman.

Zimbra really doesn’t like to have anything else running on its server. I understand this. It’s not just a control-freak thing, it means they can build and create an easily-installed turnkey box.

This has led to two strategies for integrating Mailman and Zimbra: the cowboy strategy of running Mailman on the Zimbra server alongside Zimbra, and the officially supported strategy of running Mailman on its own separate server, with a Zimbra-provided API hook for better integration.

I knew I was going to have another server for my web services, so I thought I would start out with the “separate server” strategy. Installing Mailman is simple enough, but then it has to be knitted carefully into an Apache web server as well as Postfix.

This quickly began to seem like the more fragile and complex of the two solutions. It would require patching Mailman and Zimbra to support Zimbra’s API to manage all of the required mailing list aliases. It also will not support multiple domains, which makes that strategy dead on arrival.

So back to the cowboy approach, knit Mailman into the Zimbra server, hopefully without making too much of an impact on Zimbra’s ecosystem.

This took some doing, but was a good project for a Sunday while watching the final round of the Masters. 

I won’t get directly into the details here. I know there will be sysadmins curious about the problems I encountered and solved. None of the “how-tos” and forum threads covered every problem, and many of them were several years old. I owe it to the community to write up as much of my problems and solutions as I can. I promise to do that in a follow-up blog post.

The final result is a thing of pride and beauty. Here’s a little of what I did.

Mailman needs to be knitted into two services: Apache (web service) and Postfix (mail service). On Zimbra’s box, Zimbra owns both of these.

The web service knit turned out to be reasonable and elegant. The solution I liked is to move the Zimbra server out of the way to another port (81 in my case).  Zimbra provides a local configuration flag to change that port that will be preserved in an upgrade. Then I installed a server-wide Apache package and configured it for Mailman in a virtual host. In another virtual host, I set up an http proxy that will forward to Zimbra’s apache instance on localhost port 81.

With this setup, the Mailman web users will find the familiar Mailman applications at lists.rs.whiteoaks.com, and the Zimbra users will find their email applications at mail.rs.whiteoaks.com.

The second knitting job requires patching Mailman into Zimbra’s owned and operated Postfix instance. There are only three spots that need to be patched in the Postfix main.cf configuration file, alias maps, virtual alias maps, and mydestination. Patching them works fine.

But here’s the rub: every time Zimbra restarts Postfix it will rebuild main.cf from its configuration. This is annoying at first, and brilliant in retrospect. By setting Zimbra’s local configuration variables, the Mailman hooks will be preserved not only through restarts, but through package upgrades of Zimbra. (Details forthcoming, I promise.)

A few days ago I began forwarding my main email inbox to the mail server on my Zimbra instance, and yesterday I migrated the mailing lists for Bridgemojo.  Both are working swimmingly.

A clean untrained Spamassassin lets through a lot of marginal junk mail, but I was delighted to learn that clicking “Junk” in Thunderbird and moving spam to the Junk folder would automatically train Spamassassin.  Likewise finding good email in the Junk folder and clicking “Not Junk” would also train Spamassassin.

I can now use “push email” from the Zimbra IMAP server on my Android phone. My old “wu” imapd servers could not support push along with a desktop email client without getting hopelessly tangled. The old servers are also developing a habit of hanging and locking up changes to the mailbox until I find the problem and kill it.

I knew this would be the hardest part of migrating to the cloud. Moving the web services should be a piece of cake. One additional 512MB server should handle those easily, especially with no mail services chewing up memory.

Before publishing this, I moved mojo.whiteoaks.com to my cloud server. Life is good.