Sunday, November 25, 2007

Postfix + DSPAM 3.8.0 + Ubuntu

I have been wrestling with my dspam configuration on Ubuntu for quite some time and think I finally got it set up the optimal way. It took building a custom modern dspam package myself, with the help of a kind soul who built a custom package for Debian etch.

I get tens of thousands of spam messages to my personal accounts each month. And there are many more going to other users at my domain. It has been getting worse recently. This primarily caused me to take more drastic action and implement realtime blackhole lists to block spam from even entering my mail system. It is absolutely stunning to see how much spam gets blocked vs. how much gets in now. I haven't calculated the stats but on a cursory look at my logs, it is well over 70% that is being dropped on the floor now.

I was having a really bad issue with dspam 3.6.8 that comes with Ubuntu. Turns out this is a very old version of dspam. 3.8.0 has been out for well before the current 7.10 release yet it is only now being looked at for inclusion in 8.04 in April 2008. Ugh. Part of the problem is that upstream Debian hasn't upgraded yet in any of their repositories -- even unstable. Reference: source/dspam/ bug/160139

Alas, I set about building my own. I found a great resource at that had binary builds for debian etch and the source packages. Rather than wrestle with redoing the work of applying 3.6.8 patches to 3.8.0, I started with this and it actually builds everything cleanly on Ubuntu 7.04 just fine.

First thing is to make sure you have all of the prerequisites for building packages and building dspam. dspam requires at least mysql, postgres, ldap, zlib and other libraries to build, as well as automake and other build tools.

sudo apt-get install build-essential
sudo apt-get build-dep dspam
Obtain the source code and debian patch:
Unpack everything and apply the patch
tar xvzf dspam_3.8.0.orig.tar.gz
gunzip dspam_3.8.0-1.1etch1.diff.gz
cd dspam-3.8.0
patch -p1 < ../dspam_3.8.0-1.1etch1.diff
Now, build everything, including the .deb packages to install. You can skip this and do debian/rules install (as root) if you want to install without packages after compiling.
chmod 755 debian/rules
fakeroot debian/rules binary
Now, install the new packages. Note there is a new dependency on a base libdspam7 package for any of the driver packages. I use mysql by the way.
cd ..
sudo dpkg -i libdspam7-drv-mysql_3.8.0-1.1etch1_i386.deb \
libdspam7_3.8.0-1.1etch1_i386.deb \
dspam_3.8.0-1.1etch1_i386.deb \
dspam-webfrontend_3.8.0-1.1etch1_all.deb \
There were some changes in the config file from 3.6.8 to 3.8.0 so I would suggest starting with the new config file and integrating your customizations. This worked the best for me, although I did forget a few settings here and there so diff is your friend. That's all it took to get upgraded. My final working DSPAM postfix configuration went through some modifications as well. I originally had DSPAM integrated as a content_filter, but that runs dspam for all incoming _and outgoing_ messages. I didn't think this would be a problem at first, but after seeing it in action it became confusing for end users. What can happen in this configuration is dspam can tag the message subject with your SPAM tag when _the recipient_ (which is often a mailing list) has dspam run on it, but then dspam is run again for each individual recipient upon delivery so can end up deciding that the message is not spam, but the subject is left alone. Thus, users receive a message tagged as spam that isn't, according to their dspam decision. I instead set up using this general guideline, which is excellent: I don't use ClamAV though so that's the major difference. I've seen so many security notices sent to Bugtraq that I'm not sure the cure is better than the disease... I wanted system-wide spam and notspam retraining aliases though, so I included another transport filter in my configuration to handle those special users first before dspam got to them:
smtpd_recipient_restrictions =
check_recipient_access hash:/etc/postfix/dspam_retrain_aliases,
check_recipient_access pcre:/etc/postfix/dspam_incoming_filter,
Then, in the dspam_retrain_aliases file I have:
[email protected]                  FILTER dspam-fp:innocent
[email protected] FILTER dspam-add:spam
These trigger the following filters in /etc/postfix/ Note: you need to set up these subdomains in DNS first! You could probably do something like this without subdomains but that's how I and others have gotten it to work.
# only allow local network to post to these entries
dspam-add unix - n n - - pipe
flags=Rhq user=dspam argv=/usr/bin/dspam --mode=toe --user [email protected] --class=spam --source=error
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o mynetworks=

# only allow local network to post to these entries
dspam-fp unix - n n - - pipe
flags=Rhq user=dspam argv=/usr/bin/dspam --mode=toe --user [email protected] --class=innocent --source=error
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o mynetworks=
I also added in some header_checks to reject emails with foreign character sets in them to block additional spams. I've been getting a ton of greek spam and other mid-east charsets it seems.
# Using this to block lots of non-US character set emails
header_checks = regexp:/etc/postfix/header_checks
And I combined several regexes from various Internet sources in there:
/^Subject:.*=\?big5\?/ REJECT No foreign character sets, please.
/^Content-Type:.*charset=.*big5/ REJECT No foreign character sets, please.
/^Subject:.*=\?euc-kr\?/ REJECT No foreign character sets, please.
/^Content-Type:.*charset=.*euc-kr/ REJECT No foreign character sets, please.
/^Subject:.*=\?gb2312\?/ REJECT No foreign character sets, please.
/^Content-Type:.*charset=.*gb2312/ REJECT No foreign character sets, please.
/^Subject:.*=\?iso-.*-jp\?/ REJECT No foreign character sets, please.
/^Content-Type:.*charset=.*iso-.*-jp/ REJECT No foreign character sets, please.
/^Subject:.*=\?koi8\?/ REJECT No foreign character sets, please.
/^Content-Type:.*charset=.*koi8-r/ REJECT No foreign character sets, please.
/^Subject:.*=\?ks_c_5601-1987\?/ REJECT No foreign character sets, please.
/^Content-Type:.*charset=.*ks_c_5601-1987/ REJECT No foreign character sets, please.
# headers with 8 special characters... spam
/[^[:print:]]{8}/ REJECT Special chars in header a no-no.
Hope this summary helps someone...

Update: Fixed the build-dep installation command above. Copy/paste error...


  1. Hello, i get this error after
    debian/rules binary
    debian/rules:346: warning: ignoring old commands for target `unpatch'
    dh_testdir: debian/control has a duplicate entry for dspam
    make: *** [configure-stamp] Error 1
    What is my problem?

  2. I just noticed an error in what I had you run there. You need to use fakeroot to run the build:
    fakeroot debian/rules binary
    (or build as root, which I don't recommend from a security perspective) Try that and see if there's a difference.
    I've updated the posting to reflect this change.

  3. Thanks for this though sadly it seems the repo you got the patches from has now moved onto the CVS version of 3.8.1. Do you still have the patches available?
    Also the apt-get install build-dep dspam line fails for me with missing package build-dep. This is on Ubuntu 7.10. Though only a month and a bit to go so hopefully 3.8.0 will makes its way into Hardy.
    Curious what problem you had with 3.6.8? Wondering if it is safe to run with it for the next couple of months.
    Many thanks

  4. Robert,
    That repository actually contains the latest 3.8.1 CVS version plus my patches. That is what I use and have been using for months with no problems.
    3.6.8 had a ton of problems with merged groups and also, per the mailing list, there were lots of bugs that were fixed in 3.8.0 and then even more afterward. 3.8.1 stable release should actually be forthcoming and that should contain all the patches, including mine.
    Even if 3.8.0 makes it into 8.04 Ubuntu, I would go with the latest 3.8.1 or CVS build unless they have incorporated the other outstanding patches. There were loads of memory leaks and just simple bugs in the database drivers that have been fixed.
    Also, I fixed the line above regarding the build-dep. Copy/paste error. That should work better for you.
    You could also use the binary packages from the repository mentioned. Regards,

  5. Thanks Jason for taking the time to answer and also making the debs available. I will move to 3.8.1 as soon as I can, certainly I have a problem with 3.6.8 ignoring user prefs set via the webui when then are virtual users.
    I need to take your patches and build a 64bit version as my VPS is using 64bit Ubuntu.

  6. Hi,
    Thanks for this guide but system keeps saying[/tmp/dspam.sock]: No such file or directory

  7. By the way also I would like to read in your blog your opinion about new cloud technologies for business deals or maybe some kind of a virtual data room software. Is this your sphere of interest? thanks in advance for your reply!