Qmail Concurrency

Admin | November 14, 2011 | Comments Off

Qmail Concurrency

Qmail Concurrency

Problem

In my last post about qmail I said that once you solve the big concurrency problem you'll end up with another problem because your mail server would create too many outgoing connections to some domains and you risk having your ips banned by those servers.

Solution

The solution is to have a way of limiting the maximum concurrency rate by domains. To do that you'll need the   qmail channels patch or write your own patch like I did ( mostly because I was unaware of the existence of the qmail channels patch )

The home page of the qmail channels patch will explain how to setup and configure qmail to limit the concurrency by a domain or group of domains.

What I like about this patch is that it allows you to set a concurrency limit for a group of domains like set 100 for all yahoo.com, yahoo.co.uk, yahoo.ca, etc .

What I don't like is that it doesn't seem to be able to set a default concurrency level for any domain. If I'm wrong please correct me, but if I'm right then this seems like a major problem for an email server that sends to a large number of addresses distributed over a large number of domains because you would have to configure concurrency limits for a lot of domains.

The ideal solution would allow you to specify a default per domain concurrency and this would apply to any domain that doesn't have a specific concurrency. For example most email servers would be ok with 5 concurrency connections from the same ip but no way for AOL (unless you're white listed and maybe not even then ) .

Another feature I would like is to be able to specify concurrency by domain's MX records or ips/group of ips assigned to the MX servers instead of the actual domain. This would ease the configuration for ISPs that host a lot of domains like rr or yahoo.

As i have done it on a FreeBSD 8.2 box, mainly follwing instruction from here.

Instalation

tcpserver ( ucspi-tcp )

cd /usr/ports/sysutils/ucspi-tcp make install clean

Daemontools

In order to make the administration of the individual mail programs easier (POP3, SMTP, etc), daemontools will be used.  In the case that a daemon/service is brought down or ended, daemontools will make sure that it is restarted within 5 seconds ( Thusly, killing your mail services would be senseless ).

cd /usr/ports/sysutils/daemontools make install clean echo 'svscan_enable="YES"' >> /etc/rc.conf

qmail with auth patch

Install qmail — this is, after all, the main program that makes up the entire server.  First thing we want to do is extract the source so we can apply the qmail auth patch.  This allows us to have a smtp server that can relay for certain people only.

cd /usr/ports/mail/qmail make extract   killall -9 sendmail make fetch all install force-enable-qmail WITH_QMAILQUEUE_PATCH=yes WITH_BIG_TODO_PATCH=yes \ WITH_BIG_CONCURRENCY_PATCH=yes WITH_OUTGOINGIP_PATCH=yes echo 'no_sendmail="YES"' >> /etc/make.conf

Mysql server

We’re gonna use this for vpopmail. Install , enable and start.

cd /usr/ports/databases/mysql51-server/ make make install make clean echo 'mysqld_enable="YES"' >> /etc/rc.conf /usr/local/etc/rc.d/mysql-server start

Binc-Imap

Binc-Imap is a IMAP server, it is specifically geared for usage with qMail, and works very well with vpopmail

cd /usr/ports/mail/bincimap/ make install

vpopmail

The tools we can use to add new accounts and easily manage domains — Remove users, add mailing lists and more. We set WITH_DOMAIN_QUOTA’s so that we can set a quota for a whole domain, and single users. Adding spamassassin with the Alex Dupre patch is a nice way to get rid of unwanted ads and whatnot. Set SPAM_THRESHOLD to any number you want. Anything that is tagged as spam and is over this amount in points, will cause the mail to be deleted. Also tell it that we want to use Mysql

cd /usr/ports/mail/vpopmail make install clean -DWITH_MYSQL -DWITH_DOMAIN_QUOTAS -DWITH_SMTP_AUTH_PATCH -DWITH_SPAMASSASSIN SPAM_THRESHOLD=10

qmailAdmin

This will enable us to administrate the users and mailing lists over HTTP.  Note: This requires you to have apache installed and running. Please remember that the paths are relative to /usr/local, so if you want it to be in /usr/home/user/cgi-bin/ you need to make the CGIBINDIR=../home/user/cgi-bin/.

cd /usr/ports/mail/qmailadmin make all reinstall CGIBINDIR=www/cgi-bin WEBDATADIR=www/data --enable-vpopmaildir=/usr/local/vpopmail/

Replace CGIBINDIR=www/cgi-bin-dist with the correct path to your CGI-BIN, relative to /usr/local, if you changed it from the standard one which is created when you install Apache. The same thing for WEBDATADIR.

qmail-conf

This needs to be installed separatelly

cd /usr/ports/mail/qmail-conf/ make make install clean rm /usr/local/etc/rc.d/qmail.sh

 

Configuring

qmail

Now lets insert some important info into some important files.  This way qmail is happy.

hostname > /var/qmail/control/me hostname | sed "s/`hostname -s`.//g" > /var/qmail/control/defaultdomain hostname | sed "s/`hostname -s`.//g" > /var/qmail/control/plusdomain

Now let’s do the actual configuring.

/var/qmail/bin/qmail-pop3d-conf /usr/local/vpopmail/bin/vchkpw qmaill /var/qmail/service/pop3d cd /var/qmail/service/pop3d/ echo ":allow" > tcp echo "200" > env/CONCURRENCY make /var/qmail/bin/qmail-smtpd-conf qmaild qmaill /var/qmail/service/smtpd cd /var/qmail/service/smtpd/ echo "200" > env/CONCURRENCY echo '127.0.0.1:allow,RELAYCLIENT=""' > tcp echo ":allow" >> tcp make cat run | sed "/qmaild/s//root/" > run.new mv run.new run cat run | sed "/\/var\/qmail\/bin\/qmail-smtpd/s//\/var\/qmail\/bin\/qmail-smtpd {vpopmail}/" > run.new mv run.new run cat run | sed "/{vpopmail}/s//\/usr\/local\/vpopmail\/bin\/vchkpw {bintrue}/" > run.new mv run.new run cat run | sed "/{bintrue}/s//\/usr\/bin\/true/" > run.new mv run.new run chmod +x run echo '#\!/bin/sh' > /var/qmail/rc echo 'exec env - PATH="/var/qmail/bin:$PATH" \' >> /var/qmail/rc echo 'qmail-start ./Maildir/ /usr/local/bin/multilog t /var/log/qmail qmaill' >> /var/qmail/rc chmod 755 /var/qmail/rc /var/qmail/bin/qmail-delivery-conf qmaill /var/qmail/service/qmail mkdir /var/log/qmail chmod 750 /var/log/qmail chown qmaill /var/log/qmail

 

Spamassassin
This is not an complete guide, and it is just an easy way to get you going. Put the following in: /usr/local/etc/mail/spamassassin/local.cf

cat << EOF > /usr/local/etc/mail/spamassassin/local.cf # How many hits before a message is considered spam. required_hits           5.0 # Whether to change the subject of suspected spam rewrite_subject         1 # Text to prepend to subject if rewrite_subject is used subject_tag             *****SPAM***** # Encapsulate spam in an attachment report_safe             2 # Use terse version of the spam report use_terse_report        0 # Enable the Bayes system use_bayes               1 #bayes_path             /usr/local/vpopmail/.spamassassin # Enable Bayes auto-learning auto_learn              1 # Enable or disable network checks skip_rbl_checks         0 use_razor2              1 use_dcc                 1 use_pyzor               1 # How long to wait for each rbl_timeout             5 razor_timeout           5 pyzor_timeout           5 check_mx_attempts       1 # We dont want to check if DNS is available, we know it is dns_available           yes # Mail using languages used in these country codes will not be marked # as being possibly spam in a foreign language. ok_languages            all # Mail using locales used in these country codes will not be marked # as being possibly spam in a foreign language. ok_locales              all EOF

Once I get time, I’ll write a guide on most of the functions, and what you can do and add to make your SA perform better. Now to get SA started, and keep it running forever (SA is known to crash, and or just die), we add it to deamontools, just as we are going to do that with qmail later on in this guide.

Modify /var/service/smtpd/run and replace -x whatever tcp.cdb with -x/usr/local/vpopmail/etc/tcp.smtp.cdb
and comment the line

/var/qmail/bin/qmail-smtpd /usr/local/vpopmail/bin/vchkpw /usr/bin/true

and replace it with

/var/qmail/bin/qmail-smtpd 2>&1

Modify /var/service/pop3d/run and replace -x whatever tcp.cdb with -x/usr/local/vpopmail/etc/tcp.smtp.cdb

 

Binc-IMAP
This is binc-imap with basic functionality. I will not bother to get a self signed SSL cert going or other stuff like that. If you need such functionality you can read the binc-imap support/help files.

cat << EOF > /usr/local/etc/bincimap/bincimap.conf /* -*- mode:java -*- */ // This program is free software; you can redistribute it and/or modify // it under the terms of the GNU General Public License as published by // the Free Software Foundation; either version 2 of the License, or // (at your option) any later version. // You should have received a copy of the GNU General Public License // along with this program; if not, write to the Free Software // Foundation, Inc., 59 Temple Street #330, Boston, MA 02111-1307, // USA. // Global configuration file for Binc IMAP. Some settings in this file // can be overrun with command line arguments. //---------------------------------------------------------------------------- Authentication { allow plain auth in non ssl = "yes", auth penalty = 4, disable starttls = "yes" } //---------------------------------------------------------------------------- Security { jail path = "/usr/local/vpopmail/domains", jail user = "vpopmail", jail group = "vchkpw" } //---------------------------------------------------------------------------- Log { type = "multilog", environment ip variable = "TCPREMOTEIP" } //---------------------------------------------------------------------------- Mailbox { depot = "IMAPdir", type = "Maildir", path = "IMAPdir", auto create inbox = "yes", auto subscribe mailboxes = "INBOX", umask = "077" } //---------------------------------------------------------------------------- Session { idle timeout = 1860, auth timeout = 60, transfer timeout = 1200, transfer buffer size = 1024 } //---------------------------------------------------------------------------- EOF chmod uog+r-wx /usr/local/etc/bincimap/bincimap.conf

For more info about this file please read /usr/local/etc/bincimap/bincimap.conf.sample. The last command is just to set the config file to read only for owner group and user, that way it can’t be changed unless you have root privileges.
Next up we have to write a little script to automatically add some directories so that binc-imap can create folders and its cache files.

Run this script everytime you added a user using qmailadmin or by using the commandline tools. It creates the .spamassassin dirs, and a IMAPdir which binc-imap uses. Now, if the user creates folders they will have to keep using IMAPdir to access those mails, but otherwise they can switch between IMAP and pop3, as both of them can access their Maildir. Ignore any errors it generates because of trying to create the same folder twice. Everything is good.

cat << EOF > /usr/local/vpopmail/domains/imapdir.sh #!/bin/sh OLDPWD=$PWD for i in `find /usr/local/vpopmail/domains/ -type d -maxdepth 1 | grep -v '^.$'\`; do cd $i for e in `find . -type d -maxdepth 1 | grep -v '^.$'`; do cd $e mkdir IMAPdir chmod 700 IMAPdir chown vpopmail:vchkpw IMAPdir cd IMAPdir ln -s ../Maildir INBOX cd .. mkdir .spamassassin chmod 700 .spamassassin chown vpopmail:vchkpw .spamassassin cd .. done cd .. done cd $OLDPWD EOF chmod +x /usr/local/vpopmail/domains/imapdir.sh

Create the run and log run files needed for spamassassin and binc-imap.

mkdir /var/spamd-service cd /var/spamd-service echo "#\!/bin/sh" > run echo "exec 2>&1" >> run echo "exec /usr/local/bin/spamd -c -v -u vpopmail -s stderr" >> run chmod +x run mkdir log cd log cat << EOF > run #!/bin/sh exec setuidgid qmaill \ multilog \ t \ \${MAXFILESIZE+"s\$MAXFILESIZE"} \ \${MAXLOGFILES+"n\$MAXLOGFILES"} \ \${PROCESSOR+"!\$PROCESSOR"} \ ./main EOF mkdir main chown qmaill:qmail main chmod +s main chmod +x run

And we also have to add an entry for Binc-Imap. It is just like all of the other services so far started from deamontools

mkdir /var/qmail/service/binc-imap cd /var/qmail/service/binc-imap/ cat << EOF > run #!/bin/sh exec 2>&1 \ envdir ./env \ sh -c ' case "\$REMOTENAME" in h) H=;; p) H=p;; *) H=H;; esac case "\$REMOTEINFO" in r) R=;; [0-9]*) R="t\$REMOTEINFO";; *) R=R;; esac exec \ softlimit \${DATALIMIT+"-d\$DATALIMIT"} \ /usr/local/bin/tcpserver \ -vD"\$H\$R" \ \${LOCALNAME+"-l\$LOCALNAME"} \ \${BACKLOG+"-b\$BACKLOG"} \ \${CONCURRENCY+"-c\$CONCURRENCY"} \ -xtcp.cdb \ -- "\${IP-0}" "\${PORT-143}" \ /usr/local/bin/bincimap-up --logtype=multilog \ --conf=/usr/local/etc/bincimap/bincimap.conf -- \ /usr/local/vpopmail/bin/vchkpw \ /usr/local/bin/bincimapd ' EOF chmod +x run mkdir env cd env touch BACKLOG CONCURRENCY DATALIMIT IP LOCALNAME MAILDIRNAME POPUPHOST REMOTEINFO REMOTENAME echo "200" > CONCURRENCY echo "500000" > DATALIMIT cd .. echo ":allow" > tcp /usr/local/bin/tcprules tcp.cdb tcp.tmp < tcp mkdir log cd log cat << EOF > run #!/bin/sh exec setuidgid qmaill \ multilog \ t \ \${MAXFILESIZE+"s\$MAXFILESIZE"} \ \${MAXLOGFILES+"n\$MAXLOGFILES"} \ \${PROCESSOR+"!\$PROCESSOR"} \ ./main EOF mkdir main chown qmaill:qmail main chmod +s main chmod +x run

Next we have to add entries so that once we start daemontools it is able to start up all the processes needed for qmail to work.

mkdir /var/service ln -s /var/qmail/service/pop3d /var/service/pop3d ln -s /var/qmail/service/smtpd /var/service/smtpd ln -s /var/qmail/service/qmail /var/service/qmail ln -s /var/spamd-service /var/service/spamd-service ln -s /var/qmail/service/binc-imap /var/service/binc-imap

Configuring vpopmail

Set-up mysql database , tables and permissions

Enter mysql:

mysql -u root -p

Create database & user :

CREATE DATABASE vpopmail; GRANT ALL PRIVILEGES ON vpopmail.* TO vpopmailuser@"hostname" IDENTIFIED BY 'password'; quit;

Set-up vpopmail mysql server settings

echo 'localhost|0|vpopmailuser|password|vpopmail' > /usr/local/vpopmail/etc/vpopmail.mysql

 

Now we need to add a domain and a user to the entire system.  We do that using the vpopmail tools installed in /usr/local/vpopmail/bin.  First we want to add a domain.  I suggest adding your computer’s hostname or any domain that is pointed at it.  Note: sub-domains allowed.

/usr/local/vpopmail/bin/vadddomain domain.tld

Since we dont want to have a system with just postmaster, since that is a bad idea, we need to add a user as follows:

/usr/local/vpopmail/bin/vadduser user1@domain.tld

For any other actions, you can take a look at the commands by going to /usr/local/vpopmail/bin and doing an ls -al.  To get more info on a command, just type the command name without any parameters and the output will then be displayed.

cd /usr/local/vpopmail/bin ./vadduser vadduser: usage: [options] email_address [passwd] options: -v (print the version) -q quota_in_bytes (sets the users quota) -s (don't rebuild the vpasswd.cdb file, faster for large sites) -c comment (sets the gecos comment field) -e standard_encrypted_password -n no_password -r generate a random password

For further reading, check out http://inter7.com/ — the creators of vpopmail.

Using qmailAdmin
Using qmailAdmin is simple and straight forward.  In any normal browser goto http://yourserversname.com/cgi-bin/qmailadmin/qmailadmin and login as postmaster, fill in the domain you want to login for and type in the password you set.  From here you can add users, edit mailing lists and a lot of other things.

daemontools
Now lets start the entire thing.  We can, for easy access, just use the same script that will startup qmail and daemontools on a restart:

/usr/local/etc/rc.d/svscan.sh start

If all goes well, you can do a “ps aux” to see if tcpserver and qmail are running.  If not then you must have done something wrong.  If you have added a user account, check to see if auth is working.  If it isn’t and you skipped the section of making qmail-smtpd run as root, then go back and make the changes.

Restarting deamontools is not the way to go about restarting your qmail, install, or restarting spamassassin. To restart deamontools:

/usr/local/etc/rc.d/svscan.sh stop /usr/local/etc/rc.d/svscan.sh start

To restart spamassassin or any other service, issue the following commands:

svc -t /var/service/spamd-service

Replace spamd-service with any other directory that is currently in /var/service/. If you would like to restart the superviser that is watching in this cause spamd, you can issue the following command:

svc -dx /var/service/spamd-service

We would also like to be able to get some stats. Preferably on the amount of uptime, or how long a service has been up and running for. There is a special command for that called svstat. Use it as so:

svstat /var/service/*

Or if you want individual uptimes:

svstat /var/service/spamd-service