User:SwifT/Wikified but not merged documents/Virtual mail HOWTO

Introduction
For most Gentoo users, a simple mail client and fetchmail will do. However, when hosting a domain a full blown MTA (Mail Transfer Agent) is needed. When hosting multiple domains, then something more robust is required to handle all of the email for the users. This system was designed to be an elegant solution to that problem.

A virtual mail system needs to be able to handle email for numerous domains with multiple users over a variety of interfaces. This presents some issues that must be addressed. For instance, what if two users on different domains want the same user name? If IMAP access and smtp-auth are being provided, how does one combine the various authentication daemons into a single system? How security provided for the numerous components that comprise the system? How will one manage it all?

This guide will provide instructions on how to set up with a mail system capable of handling mail for as many domains as the system hardware can handle, supports virtual mail users that do not require shell accounts, has domain specific user names, can authenticate web, IMAP, SMTP, and POP3 clients against a single database, utilizes SSL for transport layer security, has a web interface, can handle mailing lists for any domain on the machine, and is controlled by a nice, central and easy MYSQL database.

There are quite a variety of ways to go about setting up a virtual mailhosting system. With so may options, another may be the best choice for your specific needs. Consider investigating http://www.qmail.org/ and http://www.exim.org/ to explore possible options.

The following packages will be used in this setup: apache, courier-imap, courier-authlib postfix, mod_php, phpmyadmin, squirrelmail, cyrus-sasl, mysql, php, and mailman.

Make sure to turn on the following USE variables in before compiling the packages:. Otherwise things will most likely have to be recompiled to get the needed support for all the protocols. Further, it is a good idea to turn off any other mail and network variables, like ipv6.

Postfix basics
Install postfix:

After postfix is installed, it can be configured. Change the following options in Remember to replace   with appropriate names.

Next change the following in. This will turn on verbose output for debugging:

{{FileBox|filename=/etc/postfix/master.cf|1=
 * 1) service type  private unpriv  chroot  wakeup  maxproc command + args
 * 2)               (yes)   (yes)   (yes)   (never) (50)

=
============================================================= smtp     inet  n       -       n       -       -       smtpd -v }}
 * 1) (Just add the "-v" after the smtpd in the following line)

Next, edit to add local aliases. There should at least be an alias for root like:.

Starting postfix for the first time:

This will install the new aliases. This step needs to be performed when aliases are updated or installed.

Now that postfix is running, fire up your favorite console mail client and send a test email. mutt can be used to send mail without leaving the terminal window. Verify that postfix is delivering mail to local users, once confirmed move on to the next step.

Courier-imap
Install courier-imap and courier-authlib:

Courier-imap configuration:


 * 1) (Change the C, ST, L, CN, and email parameters to match your server.)
 * 2) NOTE: C is a 2 char country code.

Start the courier services you need:

Start up a favorite mail client and verify that all started connections work for receiving and sending mail. Of course, you won't be able to log on to any of the services because authentication hasn't been configured yet, but it is wise to check if the connections themselves work or not.

Now that the basics work, we're going to do a whole bunch of stuff at once to get the rest of the system running. Again, please verify that what we've installed already works before progressing.

Cyrus-sasl
Next we're going to install cyrus-sasl. Sasl is going to play the role of actually passing your auth variables to courier-auth, which will in turn pass that information to mysql for authentication of smtp users. For this howto, we'll not even try to verify that sasl is working until mysql is set up and contains a test user. Which is fine since we'll be authenticating against mysql in the end anyway.

Configuring and installing the cyrus-sasl ebuild:

Next, edit

SASLAUTHD_OPTS="${SASLAUTH_MECH} -a rimap -r" SASLAUTHD_OPTS="${SASLAUTHD_OPTS} -O localhost" }}
 * 1) nano -w /etc/conf.d/saslauthd
 * 1) /etc/init.d/saslauthd start

SSL Certs for Postfix and Apache
Next we're going to make a set of ssl certificates for postfix and apache.

If the variables are not already present, just add them in a sensible place.

Now we do the same thing for apache:

Just leave the resulting certificates here for now. We'll install them after Apache is installed.

Adding SSL and SASL support to Postfix
Now edit the postfix config's to make it aware of the new sasl and ssl capabilities. Add the following parameters to the end of the file where they will be easy to find.


 * 1) (The broken_sasl_auth_clients option and the login auth method are for outlook and outlook express only and are undocumented. Isn't having to hack software for stupid, broken, M$ BS great? smtpd_sasl_local_domain appends a domain name to clients using smtp-auth. Make sure it's blank or your user names will get mangled by postfix and be unable to auth.)

smtpd_recipient_restrictions = permit_sasl_authenticated, permit_mynetworks, reject_unauth_destination

smtp_use_tls = yes smtp_tls_note_starttls_offer = yes smtpd_use_tls = yes smtpd_tls_key_file = /etc/postfix/newkey.pem smtpd_tls_cert_file = /etc/postfix/newcert.pem smtpd_tls_CAfile = /etc/postfix/cacert.pem smtpd_tls_loglevel = 3 smtpd_tls_received_header = yes smtpd_tls_session_cache_timeout = 3600s tls_random_source = dev:/dev/urandom
 * 1) (The next two options enable outgoing encryption.)
 * 1) smtpd_tls_auth_only = yes


 * 1) (smtpd_tls_auth_only is commented out to ease testing the system. You can turn this on later if you desire.)

}}
 * 1) postfix reload

Now we're going to verify that the config's we added were picked up by postfix. For this we are going to use  (provided by for instance   ) although you can also use   (provided by   ):

Verifying sasl and tls support

Verify that the above AUTH and STARTTLS lines now appear in your postfix install. As I said before, as it stands now AUTH will not work. that's because sasl will try to auth against it's sasldb, instead of the shadow file for some unknown reason, which we have not set up. So we're going to just plow through and set up mysql to hold all of our auth and virtual domain information.

The vmail user
Before we set up our virtual mailhosting environment, we create a functional user under which the virtual mailboxes will be hosted. For clarity's sake we will call this vmail :

Adding the vmail user

So now you've set up the vmail account. You can create multiple accounts if you want (to keep some structure in your set of virtual mail accounts). The user id, group id and home dirs are referenced in the MySQL tables.

Next to the user account we also need to create the location where the mailboxes will reside:

Creating mailboxes

MySQL
Next we're going to install and configure MySQL. You'll need the genericmailsql.sql dumpfile for this step.

Installing and configuring MySQL

Your new database has default values and tables set up for two domains. The following tables are included:


 * alias - local email alias and mailman alias information.
 * relocated - relocated user email address maps
 * transport - default mail transport information for all domains you are hosting
 * users - all user account information
 * virtual - virtual domain email alias maps

alias table sample

user table sample

The values of the  uid and gid should be those of the   user and group.

transport table sample

virtual table sample

Apache and phpMyAdmin
Next we'll set up apache and add an interface to interact with the database more easily.

Setting up apache and phpmyadmin

There are plenty of guides out there about how to set up apache with php, including guides provided by the Gentoo PHP Project. There are also numerous posts on http://forums.gentoo.org detailing how to solve problems with the installation. So, that said, we're not going to cover it here. Set up the apache and php installs, then continue with this howto. Now, a word for the wise: .htaccess the directory that you put phpmyadmin in. If you do not do this, search engine spiders will come along and index the page which in turn will mean that anyone will be able to find your phpmyadmin page via google and in turn be able to come change your database however they want which is BAD! There are many howtos on this including: http://www.csoft.net/docs/micro/htaccess.html.en.

Now we're going to install the Apache certificates we made previously. The Apache-SSL directives that you need to use the resulting cert are:


 * SSLCertificateFile /path/to/certs/new.cert.cert
 * SSLCertificateKeyFile /path/to/certs/new.cert.key

Install Apache SSL certificates

Next, configure phpMyAdmin.

Configuring phpMyAdmin

Now enter the phpmyadmin page and browse the tables. You'll want to add in your local aliases, edit your user table to add a test user, and change your transport table to add information about your domains. The default values supplied with the dumpfile should be a sufficient guide to what values need to go where. Make sure that if you put information in the database that it is accurate. For instance, make sure the local user's home dir exists and that the correct uid/gid values are in place. The maildirs should be created automatically by postfix when the user receives their first email. So, in general, it's a good idea to send a "Welcome" mail to a new user after you setup their account to make sure the .maildir gets created.

Configuring MySQL Authentication and vhosts
Next we'll reconfigure our authentication to use the mailsql database in courier-imap and postfix. In all of the following examples, replace  with the password you set for the mailsql mysql user.

Configuring authentication

We're almost there, I promise! Next, set up the rest of the necessary configs for postfix to interract with the database for all its other transport needs. Remember to replace each value with the name of your own user, user id, password, alias, email address, and so on.

/etc/postfix/mysql-aliases.cf

/etc/postfix/mysql-relocated.cf

/etc/postfix/mysql-transport.cf (optional)

/etc/postfix/mysql-virtual-gid.cf (optional)

/etc/postfix/mysql-virtual-maps.cf

/etc/postfix/mysql-virtual-uid.cf (optional)

/etc/postfix/mysql-virtual.cf

Lastly, edit one more time.

/etc/postfix/main.cf

For security reasons you should change the permissions of the various :

Changing file permission

As of Postfix 2.0.x, there were a number of significant changes over the 1.1.x release. Notably the transport, virtual-gid, and virtual-uid tables are no longer necessary. The tables are still included if you wish to use them.

Make postfix reload its tables

Now, if all went well, you should have a functioning mailhost. Users should be able to authenticate against the sql database, using their full email address, for pop3, imap, and smtp. I would highly suggest that you verify that everything is working at this point. If you run into problems (with as many things as this setup has going on, it's likely that you will) check the troubleshooting section of this howto.

Squirrelmail
Install squirrelmail

Mailman
Last step: mailman. The new version of mailman has very nice virtual domain support, which is why I use it, not to mention it's really a great package. To get this package installed and working correctly for virtual domains is going to require a bit of hacking. I really recommend reading all of the mailman documentation, including README.POSTFIX.gz, to understand what's being done here.

Install mailman

Setting defaults: Mailman/Defaults.py

mailman config: mm_cfg.py

And last but not least

Adding mailman alias support to postfix

You should now be able to setup mailing lists for any domain on your box. Last note on this, make sure you run all mailman commands as the user mailman or else the permissions will be wrong and you'll have to fix them. Read the mailman doc's for more information on setting up and managing mailman lists.

Content Filtering and Anti-Virus
For content filtering and Anti-Virus, please consult our mail filtering gateway guide.

Wrap Up
Ok, you're all set, edit and turn off verbose mode for production use. You'll probably also want to add the services to your startup routine to make sure everything comes back up on a reboot. Make sure to add all the services you're using - apache, mysql, saslauthd, postfix, courier-imapd, courier-imapd-ssl, courier-pop3d, and courier-pop3d-ssl are all up to your decision on what access you want to provide. I generally have all the services enabled.

Wrap up

Have fun!

Introduction
Troubleshooting: This is a short troubleshooting guide for the set up we've detailed how to install here. It is not exhaustive, but meant as a place to get you started in figuring out problems. With a complicated setup such as this, it's imperative that you narrow down the problem to the particular component that is malfunctioning. In general I do that by following a few steps. Start from the base of the system and work your way up, ruling out components that work along the way until you discover which component is having the problem.

Step 1: Check your config files
Typos are killers, especially when dealing with authentication systems. Scan your config's and mailsql database for typo's. You can debug all you want, but if you're not passing the right information back and forth to your mail system, it's not going to work. If you make a change to a config file for a service, make sure you restart that service so that the config change gets picked up.

How to restart a service

Step 2: Are all the necessary services actually running?
If it's not running, start it up. It's awful hard to debug a service that isn't running. Sometimes a service will act like it's started but still not function. Sometimes, when a bad config is used, or a bad transmission comes into a mail component, the service will hang and keep the port from being used by another process. Sometimes you can detect this with netstat. Or, if you've been at it awhile, just take a break and reboot your box in the meantime. That will clear out any hung services. Then you can come back fresh and try it again.

Checking the status of a service

Step 3: Are all the service using the current config's?
If you've recently made a change to a config file, restart that service to make sure it's using the current version. Some of the components will dump their current config's to you, like postfix.

Some services can dump their current config

Step 4: Check the logs
Repeat after me, logs are my friend. My next troubleshooting stop is always the logs. Sometimes it's helpful to try a failed operation again then check the logs so that the error message is right at the bottom (or top depending on your logger) instead of buried in there somewhere. See if there is any information in your log that can help you diagnose the problem, or at the very least, figure out which component is having the problem.

Checking the logs

You may also find the debug_peer parameters in main.cf helpful. Setting these will increase log output over just verbose mode.

adding debug_peer support

Step 5: Talk to the service itself
SMTP, IMAP, and POP3 all respond to telnet sessions. As we've seen earlier when we verified postfix's config. Sometimes it's helpful to open a telnet session to the service itself and see what's happening.

Connect to a service with telnet

Step 6: Sometimes only the big guns will give you the information you need: strace
You should have this installed anyway. This is an invaluable tool for debugging software. You can start commands from the command line with strace and watch all the system calls as they happen. It often dumps a huge amount of information, so you'll either need to watch it realtime as you retry a failed transaction with the mail system, or dump the output to a file for review.

Using strace

Step 7: Research
Once you have the information, if you can diagnose and fix the problem, great! If not, you'll probably need to go digging on the net for information that will help you fix it. Here's a list of sites you can check to see if your error has already been resolved. There's also a really good howto on setting up smtp-auth which contains some great debugging ideas.


 * http://forums.gentoo.org/ - Great forums for gentoo users
 * http://bugs.gentoo.org/ - Bugs database for gentoo - great place to look for specific errors
 * http://postfix.state-of-mind.de/ - smtp-auth howto
 * http://marc.theaimsgroup.com/?l=postfix-users - Postfix mailing lists - searchable
 * http://sourceforge.net/mailarchive/forum.php?forum_id=6705 - Courier-imap mailing list archives - not searchable
 * http://www.google.com/ - If all else fails, there's always google, which has never failed me
 * I also spend a lot of time on #gentoo . IRC is a great place to go for help.

Acknowledgements
We would like to thank the following authors and editors for their contributions to this guide:


 * Ken Nowack
 * Ezra Gorman
 * Zack Gilburd
 * Scygro
 * Sven Vermeulen
 * nightmorph