Mailfiltering Gateway/ru

This guide is step-by-step guide for installing spam fighting technologies for Postfix. Among them Amavisd-new using Spamassassin and ClamAV, greylisting and SPF.

Введение
Данная статья описывает пошаговую инструкцию по установке шлюза по фильтрации почты на спам и вирусы. Если у вас всего один сервер, вы можете легко адаптировать информацию данной статьи под него.

Общая картина
Данный документ описывает, как настроить шлюз для фильтрации спама в почтовом трафике со множеством доменов. Такой сервер можно запускать перед настоящими почтовыми серверами, например, Microsoft Exchange или Lotus Notes.

Для данной статьи были выбраны приложения с читаемой конфигурацией и хорошими отзывами о безопасности. Почтовый сервер - Postfix, который достаточно безопасен и который можно достаточно просто правильно настроить. Обычно Postfix проверяет входящую почту на порту 25. При получении, он будет перенаправлять ее Amavisd-new на порт 10024. Amavisd-new затем будет отфильтровывать почту через различные фильтры, а затем отправит ее обратно Postfix по порту 10025, после чего Postfix перенаправить почту следующему почтовому серверу.

Amavisd-new это фреймворк по фильтрации содержимого, использующий приложения-помощники для фильтрации на вирусы и фильтрации на спам. В данной настройке мы будем использовать два приложения-помощника - ClamAV для отфильтровывания писем с вирусами и Spamassassin для отфильтровывания спама. Spamassassin сам по себе может служить еще одним слоем фильтрации содержимого, и использовать приложения-помощники, как-то Vipul's Razor2 и DCC.

В отличие от многих других технологий борьбы со спамом, вроде RBL, Spamassassin не просто принимает или отбрасывает данное письмо, судя по одному единственному тесту. Он использует множество внутренних тестов и внешних приложений-помощников, чтобы вычислить очки спама для каждого проходящего письма. Эти очки основаны на следующих тестах:


 * Байесова фильтрация
 * Статические правила на основе регулярных выражений
 * Распределенные сети:
 * RBLs
 * Razor2
 * Pyzor
 * DCC

Первая часть (с 1 по 4 главу) данной статьи опишет базовую настройку почтового шлюза с фильтрацией. Следующие главы могут реализовываться по отдельности, без зависимости между главами. Эти главы опишут, как


 * setup special IMAP folders for learning of the Bayesian filter and for delivery of false positives
 * setup greylisting with Postfix
 * setup Amavisd-new to use a MySQL backend for user preferences
 * setup Spamassassin to use a MySQL backend for AWL and Bayes data

A planned fifth part will contain various tips regarding performance and things you may want to know (running chrooted, postfix restrictions, etc.).

Приготовления
Before you start make sure that you have a working Postfix installation where you can send and receive mails also you need a backend mailserver. If you're not experienced with setting up Postfix it might quickly become too complicated if all should be set up at once. If you need help you can find it in the excellent Complete Virtual Mail Server in the Gentoo Wiki.

Установка необходимых программ
We start out by installing the most important programs: Amavisd-new, Spamassassin and ClamAV.

Настройка DNS
While the programs are emerging fire up another shell and create the needed DNS records.

Start out by creating a  record for the mail gateway and an   record for the next destination.

Setting up DNS

Открываем файрвол
In addition to allowing normal mail traffic you have to allow a few services through your firewall to allow the network checks to communicate with the servers.

Razor использует пинги, чтобы определить ближайшие сервера.

Конфигурация Postfix
First we have to tell  to listen on port 10025 and we remove most of the restrictions as they have already been applied by the   instance listening on port 25. Also we ensure that it will only listen for local connections on port 10025. To accomplish this we have to add the following at the end of

Changing the master.cf file

The file tells the postfix master program how to run each individual postfix process. More info with.

Next we need the main  instance listening on port 25 to filter the mail through   listening on port 10024.

We also need to set the next hop destination for mail. Tell Postfix to filter all mail through an external content filter and enable explicit routing to let Postfix know where to forward the mail to.

Postfix has a lot of options set in. For further explanation of the file please consult  or the same online Postfix Configuration Parameters.

The format of the file is the normal Postfix hash file. Mail to the domain on the left hand side is forwarded to the destination on the right hand side.

After we have edited the file we need to run the  command. Postfix does not actually read this file so we have to convert it to the proper format with. This creates the file. There is no need to reload Postfix as it will automatically pick up the changes.

If your first attempts to send mail result in messages bouncing, you've likely made a configuration error somewhere. Try temporarily enabling  while you work out your configuration issues. This prevents postfix from bouncing mails on delivery errors by treating them as temporary errors. It keeps mails in the mail queue until  is disabled or removed.

Once you've finished creating a working configuration, be sure to disable or remove  and reload postfix.

Конфигурация Amavisd-new
Amavisd-new is used to handle all the filtering and allows you to easily glue together severel different technologies. Upon reception of a mail message it will extract the mail, filter it through some custom filters, handle white and black listing, filter the mail through various virus scanners and finally it will filter the mail using SpamAssassin.

Amavisd-new и сам имеет некоторое число дополнительных возможностей:


 * it identifies dangerous file attachments and has policies to handle them
 * per-user, per-domain and system-wide policies for:
 * whitelists
 * blacklists
 * spam score thresholds
 * virus and spam policies

Apart from  and   we will run all applications as the user.

Отредактируйте следующие строки в

Create a quarantine directory for the virus mails as we don't want these delivered to our users.

Конфигурация ClamAV
As virus scanner we use ClamAV as it has a fine detection rate comparable with commercial offerings, it is very fast and it is Open Source Software. We love log files, so make  log using   and turn on verbose logging. Also do not run  as. Now edit

ClamAV comes with the  deamon dedicated to periodical checks of virus signature updates. Instead of updating virus signatures twice a day we will make  update virus signatures every two hours.

Start  with   using the init scripts by modifying.

В конце, укажите в файле новое положение сокета.

Конфигурация Vipul's Razor
Razor2 is a collaborative and distributed spam checksum network. Install it with  and create the needed configuration files. Do this as user  by running   followed.

Temporarily set amavis' shell to bash:

Reset the shell to /bin/false:

Configuring Distributed Checksum Clearinghouse (dcc)
Like Razor2, dcc is a collaborative and distributed spam checksum network. Its philosopy is to count the number of recipients of a given mail identifying each mail with a fuzzy checksum.

Конфигурация Spamassassin
Amavis is using the Spamassassin Perl libraries directly so there is no need to start the service. Also this creates some confusion about the configuration as some Spamassassin settings are configured in and overridden by options in.

Every good rule has good exceptions as well
Once mail really starts passing through this mail gateway you will probably discover that the above setup is not perfect. Maybe some of your customers like to receive mails that others wouldn't. You can whitelist/blacklist envelope senders quite easily. Uncomment the following line in.

In the file you put complete email addresses or just the domian parts and then note a positive/negative score to add to the spam score.

While waiting for a better method you can add the following to to bypass spam checks for   and   mailboxes.

By pass spam filters for all postmaster and abuse mails

Добавляем больше правил
If you want to use more rules provided by the SARE Ninjas over at the SpamAssassin Rules Emporium you can easily add and update them using the  mechanism included in Spamassassin.

A brief guide to using SARE rulesets with  can be found here.

Тестирование установки
Now before you start  you can manually verify that it works.

Now you have updated virus definitions and you know that is working properly.

Test freshclam and amavisd from the cli and amavisd testmails. Start  and   with the following commands:

If everything went well  should now be listening for mails on port 25 and for reinjected mails on port 10024. To verify this check your log file.

Now if no strange messages appear in the log file it is time for a new test.

Use  to manually connect to   on port 10024 and   on port 10025.

Add  and   to the   runlevel.

Создаем пользователя spamtrap (ловушка для спама)
Create the spamtrap account and directories.

Give the spamtrap user a sensible password.

If you manually want to check some of the mails to ensure that you have no false positives you can use the following  recipe to sideline spam found into different mail folders.

Создаем .procmailrc
Now make sure that Postfix uses  to deliver mail.

Создаем каталоги для почты
Now we will create shared folders for ham and spam.

Amavisd-new needs to be able to read these files as well as all mailusers. Therefore we add all the relevant users to the mailuser group along with amavis.

This makes the spam and ham folders writable but not readable. This way users can safely submit their ham without anyone else being able to read it.

Then run the following command as the  user:

Добавляем задачи cron
Now run  to edit the amavis crontab to enable automatic learning of the Bayes filter every hour.

Изменяем amavisd.conf
Now modify amavis to redirect spam emails to the  account and keep spamheaders.

Очистка
We don't want to keep mail forever so we use  to clean up regularily. Emerge it with. Only  is able to run   so we have to edit the   crontab.

Введение
Greylisting is one of the newer weapons in the spam fighting arsenal. As the name implies it is much like whitelisting and blacklisting. Each time an unknown mailserver tries to deliver mail the mail is rejected with a try again later message. This means that mail gets delayed but also that stupid spam bots that do not implement the RFC protocol will drop the attempt to deliver the spam and never retry. With time spam bots will probably adjust, however it will give other technologies more time to identify the spam.

Postfix 2.1 come with a simple Perl greylisting policy server that implements such a scheme. However it suffers from unpredictable results when the partition holding the greylisting database run out of space. There exists an improved version that do not suffer this problem. First I will show how to install the builtin greylisting support that come with Postfix and then I will show how to configure the more robust replacement.

Простой грейлистинг
We need the file but unfortunately the ebuild does not install it as default.

Now we have the file in place we need to create the directory to hold the greylisting database:

Настройка грейлистинга
Now that we have all this ready all that is left is to add it to the postfix configuration. First we add the necessary information to the :

Изменяем master.cf для использования грейлистинга

The postfix spawn daemon normally kills its child processes after 1000 seconds but this is too short for the greylisting process so we have to increase the timelimit in :

We don't want to use greylisting for all domains but only for those frequently abused by spammers. After all it will delay mail delivery. A list of frequently forged MAIL FROM domains can be found online. Add the domains you receive a lot of spam from to :

Format of sender_access

If you want a more extensive list:

Now we only have to initialize the database:

Now the setup of simple greylisting is complete.

Конфигурация улучшенного грейлистинга с помощью postgrey
You can install the enhanced greylisting policy server with a simple  :

After installing  we have to edit. Changes are almost exactly like the built in greylisting.

Finally, start the server and add it to the proper runlevel.

Введение
SPF allows domain owners to state in their DNS records which IP addressess should be allowed to send mails from their domain. This will prevent spammers from spoofing the.

First domain owners have to create a special  DNS record. Then an SPF-enabled MTA can read this and if the mail originates from a server that is not described in the SPF record the mail can be rejected. An example entry could look like this:

Пример записи SPF

The  means to reject all mail by default but allow mail from the ,   and   DNS records. For more info consult further resources below.

Spamassassin 3.0 has support for SPF, however it is not enabled by default and the new policy daemon in Postfix supports SPF so let's install SPF support for Postfix.

Приготовления
First you have to install Postfix 2.1 as described above. When you have fetched the source grab the with:

This Perl script also needs some Perl libraries that are not in portage but it is still quite simple to install them:

Now that we have everything in place all we need is to configure Postfix to use this new policy.

Now add the SPF check in. Properly configured SPF should do no harm so we could check SPF for all domains:

Настройка MySQL
For large domains the default values you can set in might not suit all users. If you configure amavisd-new with MySQL support you can have individual settings for users or groups of users.

Now that the database is created we'll need to create the necessary tables. You can cut and paste the following into the mysql prompt:

MySQL table layout

If you wish to use whitelisting and blacklisting you must add the sender and receiver to  after which you create the relation between the two e-mail addresses in   and state if it is whitelisting  or blacklisting.

Now that we have created the tables let's insert a test user and a test policy:

Create test user and test policy

This inserts a test user and a Test policy. Adjust these examples to fit your needs. Further explanation of the configuration names can be found in.

Конфигурация amavisd для использования MySQL
Now that MySQL is ready we need to tell amavis to use it:

Конфигурация Spamassassin для использования MySQL
As of Spamassassin 3.0 it is possible to store the Bayes and AWL data in a MySQL database. We will use MySQL as the backend as it can generally outperform other databases. Also, using MySQL for both sets of data makes system management much easier. Here I will show how to easily accomplish this.

First start out by creating the new MySQL user and then create the needed tables.

Now that the database is created we'll create the necessary tables. You can cut and paste the following into the mysql prompt:

MySQL table layout

Конфигурация Spamassassin для использования MySQL
If you have an old Bayes database in the DBM database and want to keep it follow these instructions:

Now give Spamassassin the required info:

Next, change its permissions for proper security:

Now all you have to do is.

Amavisd-new
To troubleshoot Amavisd-new start out by stopping it with  and then start it manually in the foreground with   and watch it for anomalies in the output.

Spamassassin
To troubleshoot Spamassassin you can filter an email through it with. To ensure that the headers are intact you can move it from another machine with IMAP.

If you want you can make get the same information and more with Amavisd-new using.

Повторяющиеся действия после установки
Some of the activities mentioned in this guide will need to be repeated after upgrades. For instance, the  in the section on  will need to be repeated after every update of amavisd-new.

Luckily, Gentoo provides you with the means to perform these steps automatically. In Hooking in the Emerge Process, the Gentoo Handbook explains how to execute tasks after installations of a particular package, like so:

Example bashrc snippet for running the mentioned chown

Получение помощи
Если вам нужна помощь, то можете зайти в список рассылки amavis-user. Перед тем, как вы зададите вопрос, попробуйте поискать в архиве списка рассылки Amavis User. Если вы не найдете там ответа, вы можете подписаться на список рассылки Amavis User

Если ваш вопрос конкретно относится к SpamAssassin, DCC, Razor или Postfix, зайдите на их домашние страницы, описанные ниже.

Для более детальной информации

 * Установка Amavisd-new
 * Документация по использованию Amavisd-new с Postfix
 * Документация по банку политик Amavisd-new
 * Документация по SQL в Spamassassin
 * Грейлистинг
 * Политики SMTPD для Postfix
 * Блокирование спамеров с помощью контроля HELO Postfix
 * Обзор
 * Подсказки от Jim Seymour по поводу Postfix Anti-UCE

Общие источники

 * Spamassassin
 * Amavisd-new
 * Кусочки документации по Amavisd-new
 * Vipuls's Razor
 * Pyzor
 * Distributed Checksum Clearinghouse
 * Maia Mailguard

Другая документация

 * Достаточно защищенный анти-спам шлюз с использованием OpenBSD, Postfix, Amavisd-new, SpamAssassin, Razor и DCC

Благодарности
Мы хотели бы поблагодарить следующих авторов и редакторов за их вклад в это руководство:


 * Sune Kloppenborg Jeppesen
 * Jens Hilligs
 * Joshua Saddler