Mailfiltering Gateway

This guide is step-by-step guide for installing spam fighting technologies for Postfix. Among them Amavisd-new using Spamassassin and ClamAV, greylisting and SPF.

Introduction
This guide describe step by step how to install a spam and virus filtering mail gateway. It is quite simple to adopt this to a single server solution.

The big picture
This document describe how to setup a spam filtering mail gateway with multiple domains. This server is meant to run in front of the mail servers actually keeping the mail accounts i.e. Microsoft Exchange or Lotus Notes.

In this setup applications with good security records and readable configuration files have been chosen. The email MTA is postfix which has a good security record and is fairly easy to setup right. Postfix will listen normally on port 25 for incoming mail. Upon reception it will forward it to Amavisd-new on port 10024. Amavisd-new will then filter the mail through different filters before passing the mail back to Postfix on port 10025 which in turn will forward the mail to the next mail server.

Amavisd-new is a content filtering framework utilizing helper applications for virus filtering and spam filtering. In this setup we will be using two helper applications one ClamAV for filtering virus mails and Spamassassin for filtering spam. Spamassassin itself can function as yet another layer of content filtering framework and utilize the helper applications Vipul's Razor2 and DCC.

Unlike many other spam fighting technologies like RBLs and others Spamassassin does not simply accept or reject a given email based on one single test. It uses a lot of internal tests and external helper applications to calculate a spam score for every mail passed through. This score is based on the following tests:


 * Bayesian filtering
 * Static rules based on regular expressions
 * Distributed and collaborative networks:
 * RBLs
 * Razor2
 * Pyzor
 * DCC

The first part (chapters 1 to 4) of the guide will describe the basic setup of a mailfiltering gateway. The next chapters can be implemented individually with no dependence between each chapter. These chapters describe how to:


 * setup special IMAP folders for learning of the Bayesian filter and for delivery of false positives
 * setup greylisting with Postfix
 * setup Amavisd-new to use a MySQL backend for user preferences
 * setup Spamassassin to use a MySQL backend for AWL and Bayes data

A planned fifth part will contain various tips regarding performance and things you may want to know (running chrooted, postfix restrictions, etc.).

Preparations
Before you start make sure that you have a working Postfix installation where you can send and receive mails also you need a backend mailserver. If you're not experienced with setting up Postfix it might quickly become too complicated if all should be set up at once. If you need help you can find it in the excellent Complete Virtual Mail Server in the Gentoo Wiki.

Installing the programs needed
We start out by installing the most important programs: Amavisd-new, Spamassassin and ClamAV.

Setting up DNS
While the programs are emerging fire up another shell and create the needed DNS records.

Start out by creating a  record for the mail gateway and an   record for the next destination.

Setting up DNS

Opening the firewall
In addition to allowing normal mail traffic you have to allow a few services through your firewall to allow the network checks to communicate with the servers.

Razor uses pings to discover what servers are closest to it.

Configuring Postfix
First we have to tell  to listen on port 10025 and we remove most of the restrictions as they have already been applied by the   instance listening on port 25. Also we ensure that it will only listen for local connections on port 10025. To accomplish this we have to add the following at the end of

Changing the master.cf file

The file tells the postfix master program how to run each individual postfix process. More info with.

Next we need the main  instance listening on port 25 to filter the mail through   listening on port 10024.

We also need to set the next hop destination for mail. Tell Postfix to filter all mail through an external content filter and enable explicit routing to let Postfix know where to forward the mail to.

Postfix has a lot of options set in. For further explanation of the file please consult  or the same online Postfix Configuration Parameters.

The format of the file is the normal Postfix hash file. Mail to the domain on the left hand side is forwarded to the destination on the right hand side.

After we have edited the file we need to run the  command. Postfix does not actually read this file so we have to convert it to the proper format with. This creates the file. There is no need to reload Postfix as it will automatically pick up the changes.

If your first attempts to send mail result in messages bouncing, you've likely made a configuration error somewhere. Try temporarily enabling  while you work out your configuration issues. This prevents postfix from bouncing mails on delivery errors by treating them as temporary errors. It keeps mails in the mail queue until  is disabled or removed.

Once you've finished creating a working configuration, be sure to disable or remove  and reload postfix.

Configuring Amavisd-new
Amavisd-new is used to handle all the filtering and allows you to easily glue together severel different technologies. Upon reception of a mail message it will extract the mail, filter it through some custom filters, handle white and black listing, filter the mail through various virus scanners and finally it will filter the mail using SpamAssassin.

Amavisd-new itself has a number of extra features:


 * it identifies dangerous file attachments and has policies to handle them
 * per-user, per-domain and system-wide policies for:
 * whitelists
 * blacklists
 * spam score thresholds
 * virus and spam policies

Apart from  and   we will run all applications as the user.

Edit the following lines in

Create a quarantine directory for the virus mails as we don't want these delivered to our users.

Configuring ClamAV
As virus scanner we use ClamAV as it has a fine detection rate comparable with commercial offerings, it is very fast and it is Open Source Software. We love log files, so make  log using   and turn on verbose logging. Also do not run  as. Now edit

ClamAV comes with the  deamon dedicated to periodical checks of virus signature updates. Instead of updating virus signatures twice a day we will make  update virus signatures every two hours.

Start  with   using the init scripts by modifying.

At last modify with the new location of the socket.

Configuring Vipul's Razor
Razor2 is a collaborative and distributed spam checksum network. Install it with  and create the needed configuration files. Do this as user  by running   followed.

Temporarily set amavis' shell to bash:

Reset the shell to /bin/false:

Configuring Distributed Checksum Clearinghouse (dcc)
Like Razor2, dcc is a collaborative and distributed spam checksum network. Its philosopy is to count the number of recipients of a given mail identifying each mail with a fuzzy checksum.

Configuring Spamassassin
Amavis is using the Spamassassin Perl libraries directly so there is no need to start the service. Also this creates some confusion about the configuration as some Spamassassin settings are configured in and overridden by options in.

Every good rule has good exceptions as well
Once mail really starts passing through this mail gateway you will probably discover that the above setup is not perfect. Maybe some of your customers like to receive mails that others wouldn't. You can whitelist/blacklist envelope senders quite easily. Uncomment the following line in.

In the file you put complete email addresses or just the domian parts and then note a positive/negative score to add to the spam score.

While waiting for a better method you can add the following to to bypass spam checks for   and   mailboxes.

By pass spam filters for all postmaster and abuse mails

Adding more rules
If you want to use more rules provided by the SARE Ninjas over at the SpamAssassin Rules Emporium you can easily add and update them using the  mechanism included in Spamassassin.

A brief guide to using SARE rulesets with  can be found here.

Testing the setup
Now before you start  you can manually verify that it works.

Now you have updated virus definitions and you know that is working properly.

Test freshclam and amavisd from the cli and amavisd testmails. Start  and   with the following commands:

If everything went well  should now be listening for mails on port 25 and for reinjected mails on port 10024. To verify this check your log file.

Now if no strange messages appear in the log file it is time for a new test.

Use  to manually connect to   on port 10024 and   on port 10025.

Add  and   to the   runlevel.

Creating the spamtrap user
Create the spamtrap account and directories.

Give the spamtrap user a sensible password.

If you manually want to check some of the mails to ensure that you have no false positives you can use the following  recipe to sideline spam found into different mail folders.

Creating .procmailrc
Now make sure that Postfix uses  to deliver mail.

Create mailfolders
Now we will create shared folders for ham and spam.

Amavisd-new needs to be able to read these files as well as all mailusers. Therefore we add all the relevant users to the mailuser group along with amavis.

This makes the spam and ham folders writable but not readable. This way users can safely submit their ham without anyone else being able to read it.

Then run the following command as the  user:

Adding cron jobs
Now run  to edit the amavis crontab to enable automatic learning of the Bayes filter every hour.

Modifying amavisd.conf
Now modify amavis to redirect spam emails to the  account and keep spamheaders.

Cleaning up
We don't want to keep mail forever so we use  to clean up regularily. Emerge it with. Only  is able to run   so we have to edit the   crontab.

Introduction
Greylisting is one of the newer weapons in the spam fighting arsenal. As the name implies it is much like whitelisting and blacklisting. Each time an unknown mailserver tries to deliver mail the mail is rejected with a try again later message. This means that mail gets delayed but also that stupid spam bots that do not implement the RFC protocol will drop the attempt to deliver the spam and never retry. With time spam bots will probably adjust, however it will give other technologies more time to identify the spam.

Postfix 2.1 come with a simple Perl greylisting policy server that implements such a scheme. However it suffers from unpredictable results when the partition holding the greylisting database run out of space. There exists an improved version that do not suffer this problem. First I will show how to install the builtin greylisting support that come with Postfix and then I will show how to configure the more robust replacement.

Simple greylisting
We need the file but unfortunately the ebuild does not install it as default.

Now we have the file in place we need to create the directory to hold the greylisting database:

Configuring greylisting
Now that we have all this ready all that is left is to add it to the postfix configuration. First we add the necessary information to the :

Modifying master.cf to use greylisting

The postfix spawn daemon normally kills its child processes after 1000 seconds but this is too short for the greylisting process so we have to increase the timelimit in :

We don't want to use greylisting for all domains but only for those frequently abused by spammers. After all it will delay mail delivery. A list of frequently forged MAIL FROM domains can be found online. Add the domains you receive a lot of spam from to :

Format of sender_access

If you want a more extensive list:

Now we only have to initialize the database:

Now the setup of simple greylisting is complete.

Configuring improved greylisting with postgrey
You can install the enhanced greylisting policy server with a simple  :

After installing  we have to edit. Changes are almost exactly like the built in greylisting.

Finally, start the server and add it to the proper runlevel.

Introduction
SPF allows domain owners to state in their DNS records which IP addressess should be allowed to send mails from their domain. This will prevent spammers from spoofing the.

First domain owners have to create a special  DNS record. Then an SPF-enabled MTA can read this and if the mail originates from a server that is not described in the SPF record the mail can be rejected. An example entry could look like this:

Example SPF record

The  means to reject all mail by default but allow mail from the ,   and   DNS records. For more info consult further resources below.

Spamassassin 3.0 has support for SPF, however it is not enabled by default and the new policy daemon in Postfix supports SPF so let's install SPF support for Postfix.

Preparations
First you have to install Postfix 2.1 as described above. When you have fetched the source grab the with:

This Perl script also needs some Perl libraries that are not in portage but it is still quite simple to install them:

Now that we have everything in place all we need is to configure Postfix to use this new policy.

Now add the SPF check in. Properly configured SPF should do no harm so we could check SPF for all domains:

Configuring MySQL
For large domains the default values you can set in might not suit all users. If you configure amavisd-new with MySQL support you can have individual settings for users or groups of users.

Now that the database is created we'll need to create the necessary tables. You can cut and paste the following into the mysql prompt:

MySQL table layout

If you wish to use whitelisting and blacklisting you must add the sender and receiver to  after which you create the relation between the two e-mail addresses in   and state if it is whitelisting  or blacklisting.

Now that we have created the tables let's insert a test user and a test policy:

Create test user and test policy

This inserts a test user and a Test policy. Adjust these examples to fit your needs. Further explanation of the configuration names can be found in.

Configuring amavisd to use MySQL
Now that MySQL is ready we need to tell amavis to use it:

Configuring Spamassassin to use MySQL
As of Spamassassin 3.0 it is possible to store the Bayes and AWL data in a MySQL database. We will use MySQL as the backend as it can generally outperform other databases. Also, using MySQL for both sets of data makes system management much easier. Here I will show how to easily accomplish this.

First start out by creating the new MySQL user and then create the needed tables.

Now that the database is created we'll create the necessary tables. You can cut and paste the following into the mysql prompt:

MySQL table layout

Configuring Spamassassin to use the MySQL backend
If you have an old Bayes database in the DBM database and want to keep it follow these instructions:

Now give Spamassassin the required info:

Next, change its permissions for proper security:

Now all you have to do is.

Amavisd-new
To troubleshoot Amavisd-new start out by stopping it with  and then start it manually in the foreground with   and watch it for anomalies in the output.

Spamassassin
To troubleshoot Spamassassin you can filter an email through it with. To ensure that the headers are intact you can move it from another machine with IMAP.

If you want you can make get the same information and more with Amavisd-new using.

Repeating tasks after installation
Some of the activities mentioned in this guide will need to be repeated after upgrades. For instance, the  in the section on  will need to be repeated after every update of amavisd-new.

Luckily, Gentoo provides you with the means to perform these steps automatically. In Hooking in the Emerge Process, the Gentoo Handbook explains how to execute tasks after installations of a particular package, like so:

Example bashrc snippet for running the mentioned chown

Getting help
If you need help a good place to go is the amavis-user mailing list. Before postting a question try searching the Amavis User mailing list archives. If you find no answer here you can subscribe to the Amavis User mailing list

If your question is specific to SpamAssassin, DCC, Razor, or Postfix, please refer to their respective home pages listed below.

For further information

 * Amavisd-new INSTALL
 * Amavisd-new Postfix README
 * Amavisd-new Policy bank documentation
 * Spamassassin SQL README
 * Greylisting
 * Postfix SMTPD_POLICY_README
 * Blocking spammers with Postfix HELO controls
 * SPF Overview
 * Jim Seymour's Postfix Anti-UCE Cheat Sheet

General resources

 * Spamassassin
 * Amavisd-new
 * Amavisd-new documentation bits and pieces
 * Vipuls's Razor
 * Pyzor
 * Distributed Checksum Clearinghouse
 * Maia Mailguard

Other howtos

 * Fairly-Secure Anti-SPAM Gateway Using OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor and DCC

Acknowledgements
We would like to thank the following authors and editors for their contributions to this guide:


 * Sune Kloppenborg Jeppesen
 * Jens Hilligs
 * Joshua Saddler