09 Sep 2010 
Support Center » Knowledgebase » Distributed Message Stores for Email Archive and Spam Quarantine
 Distributed Message Stores for Email Archive and Spam Quarantine
Solution

Email Store using ESMTP and Internal Queue

Introduction

 MPP has a distributed model for storing spam quarantines and archives that is designed for larger environments.  In our distributed model remote instances of MPP make quarantine/archive decisions based on your configuration and forward the email for storage via ESMTP to a centralized instance of MPP.  The centralized  instance of MPP maintains connections to the quarantine/archive message store and manages writes to the system.   This is advantageous because it removes direct database connections from instances of MPP that are involved in the flow of email, thus increasing throughput for busy systems and isolating failures in all environments.  Our distributed architecture is compatible with all message store types.

“Email Store" features refer to a set of functionality that enables two instances of MPP to work as quarantine/archive client/server. In such configuration first instance scans incoming email and decides whether to send it to quarantine/archive. If it decides so it sends email to second instance of MPP via network socket. Second instance puts email to Internal Queue on disk and replies with success to first one. The it retrieves queued email, processes and submits email to database. The key feature of this configuration is that second instance queues email and then processes it. In that way client instance doesn't wait for potentially “heavy�? processing and submission procedures so their scanning tasks runs smoothly. Email Store feature should be effective when client (or multiple clients) and server are on different machines.

Use-case View

This section contains brief description of the features from usage point of view. Entities referenced in this section are described in “Structural View�? section.

Client Side of Email Store

Client side of Email Store is configured by specifying special URI's for QUARANTINE and ARCHIVE options. Client uses "Email Store Protocol" to communicate with server.

Server Side of Email Store

Server side of Email Store is configured by enabling:

- Email Store Protocol as an input protocol

- Internal Queue

- Quarantine/Archive

Structural View

URI's for QUARANTINE and ARCHIVE options

Client MPP instance could use “smtp�? URI's for QUARANTINE and ARCHIVE as follow:

smtp://inet:10027@192.168.10.1
smtp://unix:/var/run/quarantine.socket

First form is used for remote communication.Second is for local communication via UNIX socket. When such URI is specified message data and scanning results will be transmitted to quarantine/archive using Email Store Protocol.

Email Store Protocol

Email Store Protocol is a ESMTP protocol with custom XMPPSCANINFO and XMPPSCANDETAIL extensions.

During extensions negotiations server reports what extensions and what extension elements it supports for each extension. This is standard mechanism but non-standard keywords for extensions.

client: EHLO client.host.com
server: 250-server.host.com
server: 250-XMPPSCANINFO HOST GROUP STORE STATE
server: 250 XMPPSCANDETAIL VIRUSINFO SPAMSCORE SPAMLEVEL HARASSSCORE

Before each MAIL command client passes scan result to server:

client: XMPPSCANINFO HOST=host.that.scanned.com GROUP=scan_profile
server: 250 Ok
client: XMPPSCANINFO STORE=Quarantine STATE=virus,spam,unauthorized
server: 250 Ok
client: XMPPSCANDETAIL VIRUSINFO Arbitrary text about infection
server: 250 Ok
client: XMPPSCANDETAIL SPAMSCORE Arbitrary text about spam
server: 250 Ok
client: XMPPSCANDETAIL SPAMLEVEL Arbitrary text about level
server: 250 Ok
client: XMPPSCANDETAIL HARASSSCORE Arbitrary text about harass
server: 250 Ok
client: XMPPSCANDETAIL REASON Arbitrary text about quarantine reason
server: 250 Ok

HOST - is the host that scanned mail.

GROUP - is scanning profile that was used.

STORE - is type (Archive, Quarantine) of storage that is targeted.

STATE - is scan state as comma-separated (without any whitespace) list of flags: virus,spam,harass,unauthorized,error,max_recursion,max_size,acl,encrypted,malformed. Other are self-explained.

Email Store Protocol as an input protocol'

The following options should be specified to forces server to use Email Store Protocol as an input protocol:

EMAIL_SERVER is “postfix�?;
EMAIL_SERVER_IN_PROTOCOL is “smtp�?;
EMAIL_SERVER_IN_PROTOCOL_EXTENSIONS contain “mailstore�?;
EMAIL_SERVER_IN_SOCKET should specify input socket address;

Server should never relay mail via its output channel. To be safe it should be configured to some dummy value:

EMAIL_SERVER_OUT_SOCKET is “unix:/non/existing/file/name�?;

Internal Queue

Please refer to Internal Queue for queue configuration.

Quarantine/Archive

Usual quarantine/archive URI's should be specified. ON_QUARANTINE_SUCCESS and ON_ARCHIVE_SUCCESS should be “discard�? to avoid unwanted relaying to server's output channel.

1. MPP quarantine server and MPP client are on different hosts

For this example, we will consider: 192.168.1.2 as our MPP scanner/quarantine client and 192.168.1.3 as MPP quarantine server using file://quarantine. Custom configuration file with Email Store / Internal Queue options enabled is available from: ftp://ftp.raeinternet.com/pub/mpp3/beta/mppd-quarantine.conf.xml

1.1. Quarantine server configuration

The following options should be defined in /usr/local/MPP/mppd.conf.xml on 192.168.1.3 host:

Bellow is a list of options required for mppd-quarantine.conf.xml:

<mppd>
 <common>
  ......
    <email_server>postfix</email_server>
    <email_server_in_protocol>smtp</email_server_in_protocol>
    <email_server_in_protocol_extensions>mailstore</email_server_in_protocol_extensions>
    <email_server_in_queue_enable>yes</email_server_in_queue_enable>
    <email_server_in_queue_media>/usr/local/MPP/working/queue</email_server_in_queue_media>
    <email_server_in_socket>inet:10027@192.168.1.3</email_server_in_socket>
    <email_server_out_socket>unix:/non/existing/file/name</email_server_out_socket>
    <email_server_in_threads_min>2</email_server_in_threads_min>
    <email_server_in_threads_max>4</email_server_in_threads_max>
    <quarantine type="primary">file:///usr/local/MPP/quarantine/%TYPE%/%RCPT_DOMAIN%/%RCPT_NAME%</quarantine>
    <log_dirlog_template>/var/log/MPP/$YEAR$$MONTH$$DAY$-quarantine.log</log_dirlog_template>
  .......
 </common>
 <group id="default">
 ...
  <actions>
   ...
    <on_quarantine_success>discard</on_quarantine_success>
    <on_archive_success>discard</on_archive_success>
   ...
  </actions>
 ...
 </group>
 ....
</mppd>
 Start: /usr/local/MPP/mppd -f /usr/local/MPP/mppd.conf.xml
 Stop: /usr/local/MPP/mppd -s
 Restart: /usr/local/MPP/mppd -r

 

1.2. Quarantine client configuration

In /usr/local/MPP/mppd.conf.xml on 192.168.1.2 host user: <quarantine type="primary">smtp://inet:10027@192.168.1.3</quarantine>

 Start: /usr/local/MPP/mppd -f /usr/local/MPP/mppd.conf.xml
 Stop: /usr/local/MPP/mppd -s
 Restart: /usr/local/MPP/mppd -r

2. MPP quarantine server and MPP client on the same host

Multiple mppd instances can be started now in the same time, each using a different configuration file and another PID file location. Configuration file and PID file location should be provided as options for mppd. Since 2 MPP instances will run on the same machine: mppd instance which would scan and act as quarantine client, and other mppd instance which would act as quarantine server.

2.1. Quarantine server configuration

Custom configuration file with 'EmailStore?'/'InternalQueue' options enabled is available from: ftp://ftp.raeinternet.com/pub/mpp3/beta/mppd-quarantine.conf.xml

Bellow is a list of options required for mppd-quarantine.conf.xml:

<mppd>
 <common>
  ......
    <email_server>postfix</email_server>
    <email_server_in_protocol>smtp</email_server_in_protocol>
    <email_server_in_protocol_extensions>mailstore</email_server_in_protocol_extensions>
    <email_server_in_queue_enable>yes</email_server_in_queue_enable>
    <email_server_in_queue_media>/usr/local/MPP/working/queue</email_server_in_queue_media>
    <email_server_in_socket>inet:10027@127.0.0.1</email_server_in_socket>
    <email_server_out_socket>unix:/non/existing/file/name</email_server_out_socket>
    <email_server_in_threads_min>2</email_server_in_threads_min>
    <email_server_in_threads_max>4</email_server_in_threads_max>
    <quarantine type="primary">file:///usr/local/MPP/quarantine/%TYPE%/%RCPT_DOMAIN%/%RCPT_NAME%</quarantine>
    <log_dirlog_template>/var/log/MPP/$YEAR$$MONTH$$DAY$-quarantine.log</log_dirlog_template>
  .......
 </common>
 <group id="default">
 ...
  <actions>
   ...
    <on_quarantine_success>discard</on_quarantine_success>
    <on_archive_success>discard</on_archive_success>
   ...
  </actions>
 ...
 </group>
 ....
</mppd>
 Create mppd-quarantine symlink: ln -s /usr/local/MPP/mppd /usr/local/MPP/mppd-quarantine 
 Start: /usr/local/MPP/mppd-quarantine -f /usr/local/MPP/mppd-quarantine.conf.xml -p /var/run/mppd-quarantine.pid
 Stop:  /usr/local/MPP/mppd-quarantine -s -p /var/run/mppd-quarantine.pid
 Restart: /usr/local/MPP/mppd-quarantine -r -p /var/run/mppd-quarantine.pid

2.2. Quarantine client configuration

MPP quarantine client will be regular mppd instance, using action 'quarantine' for virus/spam using "smtp://" protocol.

<quarantine type="primary">smtp://inet:10027@127.0.0.1</quarantine>

 Start: /usr/local/MPP/mppd -f /usr/local/MPP/mppd.conf.xml
 Stop: /usr/local/MPP/mppd -s
 Restart: /usr/local/MPP/mppd -r



Article Details
Article ID: 20
Created On: 23 Jan 2008 03:20 AM

 This answer was helpful  This answer was not helpful

 User Comments Add a Comment
 Back
 Login [Lost Password] 
Email:
Password:
Remember Me:
 
 Search
 Article Options
 Add Comment
 Print Article
 PDF Version
 Email Article
 Add to Favorites
Home | Register | Submit a Ticket | Knowledgebase | Downloads
Language: