Fighting spam with pf
This article was originally published at ONLamp as
Greylisting with PF.
This article will show you how I am using
PF (the the Packet Filter from the
OpenBSD project) and spamd (also
from OpenBSD) to implement greylisting and greatly reduce incoming spam. This
solution is completely MTA agnostic. You do not have to make any changes to
your existing mail server configuration to use spamd/pf. In fact, you can easily
use pf and spamd to guard any SMTP mail server. Yes, even MS Exchange.
I first read about PF when reading
Micheal Lucas‘s
Absolute OpenBSD.
If you have never read one of his books, I urge you to do so. His writing
style is very clear and easy to follow. In the book,
there are about thirty pages on PF, yet there is so much packed into that short
chapter that I knew then and there that one day I’d start using PF. That day
came a few weeks ago. My gateway at home has been happily running PF since early
October. More important is my happiness. One of the best features of any software
product is simplicity of use. PF is simple to get started with. And easy to extend
once you need the more advanced features.
A few weeks after implementing PF on my home gateway, I attended
NYCBSDCon 2006 and listened to Bob Beck’s
talk on spamd. It was so easy
that I just had to try it. So I did.
On a related note, I’ll also give you a short introduction to OS fingerprinting
and how you can use pf to block/pass packets based up on the sending OS.
TLS – be aware
If you are planning to use TLS (transport layer security) on your mail server,
be aware that spamd has no TLS. If your MTA requires TLS, or communicates
with other MTAs that require TLS, then you will probably have to whitelist
those MTAs and perhaps make TLS optional on your MTA.
A bit how greylisting works
spamd and PF work together. PF will direct any known good clients directly to
the mail server. Similarly, known bad clients will be sent to the tarpit, a slow
responding mail server, which does not add much load to your system. Finally,
new clients, not known to be either good or bad, are asked to try again later.
With the above, you have three lists of clients:
- whitelist – known good clients
- blacklist – known bad clients
- greylist – we don’t know if they are good or bad yet, but we will soon decide.
The key to greylisting is knowing that good and kind mail servers do not mind
being politely asked to come back later. This is part of the STMP protocol.
Spammers, however, are cheap and nasty. They don’t like this. They probably
won’t come back later. Why? Queue management is tricky. If you’re sending to
millions of email addresses, why worry about a few messages that you cannot
deliver? Just skip along to the next one. Spammers work on volume. We
exploit that characteristic with greylisting.
Implementing PF
My gateway was running FreeBSD
6.2 PRELREASE. PF is also available on DragonflyBSD, NetBSD, and OpenBSD
(of course!). The FreeBSD Handbook
has a good Introduction to PF.
There are several ways to enable PF, including compiling it into the kernel, or
loading the module. I added the following options to my /etc/rc.conf
file to ensure PF starts up at boot time. It also starts the logging daemon.
pf_enable="YES" pflog_enable="YES" pf_rules="/etc/pf.rules"
NOTE that I have chosen a non-default value for my PF rules. The default value,
as found in /etc/default/rc.conf is /etc/pf.conf. To avoid
any merge conflicts with mergemaster(8), I chose to use a different
file name. The default install comes with many fine examples in
/etc/pf.conf and I urge you to read them.
Designing your PF rule set is beyond the scope of this article. The OpenBSD
project has a good example.
Enabling PF
The primary interface between PF and the outside world is pfctl.
To load PF on a running system, issue this command:
# kldload pf # kldstat Id Refs Address Size Name 1 8 0xc0400000 6721fc kernel 2 1 0xc0a73000 58554 acpi.ko 3 1 0xc4eb5000 16000 linux.ko 4 1 0xc5e20000 2d000 pf.ko
Now that you have PF loaded, let’s look at what this gives you. The following
command shows you all the filter parameters:
# pfctl -s all No ALTQ support in kernel ALTQ related functions disabled FILTER RULES: INFO: Status: Disabled Debug: None Hostid: 0x595cedd1 State Table Total Rate current entries 0 searches 0 0.0/s inserts 0 0.0/s removals 0 0.0/s Counters match 0 0.0/s bad-offset 0 0.0/s fragment 0 0.0/s short 0 0.0/s normalize 0 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 0 0.0/s proto-cksum 0 0.0/s state-mismatch 0 0.0/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s TIMEOUTS: tcp.first 120s tcp.opening 30s tcp.established 86400s tcp.closing 900s tcp.finwait 45s tcp.closed 90s tcp.tsdiff 30s udp.first 60s udp.single 30s udp.multiple 60s icmp.first 20s icmp.error 10s other.first 60s other.single 30s other.multiple 60s frag 30s interval 10s adaptive.start 0 states adaptive.end 0 states src.track 0s LIMITS: states hard limit 10000 src-nodes hard limit 10000 frags hard limit 5000
Notice that the status indicates PF is disabled. To enable PF, issue this
command:
# pfctl -e No ALTQ support in kernel ALTQ related functions disabled pf enabled
There are no filter rules loaded. To test your rules, without loading them,
try this command:
pfctl -n -f /etc/pf.rules
Any syntax errors will be brought to your attention.
The following loads the rules:
pfctl -f /etc/pf.rules
Read the man page for more -s options. You can pull out NAT,
RDR, filtering, etc.
Very simple rules
This section shows very simple rules. It is a cut down version of what I use
at home. fxp0 faces my ISP. fxp1 talks to my home network. This setting in
/etc/rc.conf allows the gateway to forward packets between the two NICs:
gateway_enable="YES"
And the rule set is:
- ext_if="fxp0"
- int_if="fxp1"
- internal_net="10.11.22.0/8"
- external_addr="m.n.o.p"
- icmp_types="echoreq"
- NoRouteIPs = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8 }"
- # machines inside
- webserver="10.55.0.23"
- # machines outside
- FISH="a.b.c.d"
- STIN="e.f.g.h"
- THEF="i.j.k.l"
- table <AllowedToSSH> { $FISH, $STIN, $THEF }
- set skip on lo0
- set skip on gif0
- scrub in all
- nat on $ext_if from $internal_net to any -> ($ext_if)
- rdr on $ext_if proto tcp from any to $external_addr port 80 -> $webserver
- # block all by default
- block all
- block in quick on $ext_if from $NoRouteIPs to any
- block out quick on $ext_if from any to $NoRouteIPs
- antispoof quick for $int_if inet
- # pass all traffic to and from the local network
- pass in on $int_if from $internal_net to any
- pass out on $int_if from any to $internal_net
- pass out on $ext_if proto tcp all modulate state flags S/SA
- pass out on $ext_if proto { udp, icmp } all keep state
- pass in on $ext_if inet proto tcp from any to $webserver port 80 flags S/SA synproxy state
- pass in quick on $ext_if inet proto tcp from <AllowedToSSH> to $external_addr \ port 22 flags S/SA keep state
A few notes on specific lines above:
- 1-2 – macros that define my external and internal NICs
- 4-5 – macros that define my internal network and my public IP address.
- 13 – my development web server
- 15-18 – machines that are allowed to SSH to my gateway
- 20 – a table of those machines
- 27 – my nat rules
- 30 – redirection of port 80 on my gateway to my webserver
- 49 – allowing those machines to SSH in
I hope that’s enough to get you started with your own ruleset.
Treating different OSes differently
Equality is a nice concept. The theory is great. The practice is not
universal. Especially when it comes to operating systems. Why
discriminate? I will answer that question in the form of a story
related to me by someone who uses OS fingerprinting at home to lessen
his domestic workload. To protect the guilty, this person will be know
as Phil.
Phil uses BSD in his everyday work. At home, his kids use BSD. Their
machines also dual-boot with Windows in case they need to do specific
things for homework etc. In general, they are to use BSD. When using
Windows, they are only allowed to use the web, nothing else.
Why do these rules exist? To lessen Phil’s workload. He doesn’t want
to be removing viruses and spamware all the time. How do you keep them
off the Windows computers? You keep the Windows computers off the
Internet.
Sure,
rules are nice, but some kids, and some adults, are known not to
follow the rules. It would be nice if there was some way to enforce
this at the firewall. Enter OS fingerprinting.
OS fingerprinting is not new. nmap,
for example, uses it. From man pf.conf:
Passive OS Fingerprinting is a mechanism to inspect nuances of a TCP connection’s
initial SYN packet and guess at the host’s operating system.
It is pf’s ability to detect the Operating System at the other end of the TCP/IP connection
that allows Phil to prevent his kids from breaking the rules. To illustrate this, I’ll try
blocking incoming queries based upon the OS you are using. On my webserver at home, I will
create a website: http://pf-fingerprinting.example.org/.
To demonstrate that the website is actually running, you will be able to browse freely to the above
URL. But if you try
http://pf-fingerprinting.example.org/:8080
you will be blocked if you are using Windows.
NOTE: I have since taken down these example webpages. Sorry. 🙂
How do I do this? I’ll show you the pf rules, but the Apache setup is outside the
scope of this article.
Redirect incoming traffic to the webserver
These rules redirect incoming traffic to from the gateway to the webserver:
rdr on $ext_if proto tcp from any to $external_addr port http -> $webserver rdr on $ext_if proto tcp from any to $external_addr port 8080 -> $webserver
As you can see port 80 (http) and port 8080 are both redirected to my webserver.
Block/pass traffic on those ports
These rules will pass or block the traffic based on port and OS:
pass in quick on $ext_if inet proto tcp from any to $webserver port http flags S/SA synproxy state block in quick on $ext_if inet proto tcp from any os windows to $webserver port 8080 pass in quick on $ext_if inet proto tcp from any to $webserver port 8080 flags S/SA synproxy state
The first line allows traffic to flow freely from my internal nic to the webserver, on port 80.
The second line blocks all traffic from any Windows machine headed towards port 8080 on my webserver.
The last line passes all traffic on port 8080. The above line contains a quick directive
so if the client OS is Windows, subsequent filter rules have no effect on the packet.
I originally wanted to redirect different OS connections to different webservers, but the OS directive
is not available on the rdr statement.
Getting spamd enabled
For FreeBSD, spamd comes as a port.
The easiest way to install it is to have a fresh copy of the FreeBSD ports tree
and issue these commands:
cd /usr/ports/mail/spamd make install clean
To enable spamd, get greylisting going, and get verbose logging, add these
entries to /etc/rc.conf:
pfspamd_enable="YES" pfspamd_flags="-g -v"
See man spamd for more details on the various options you can specify.
If you are using greylisting, you also need to issue this command:
mount -t fdescfs fdescfs /dev/fd
This mount allows spamlogd to update the spamd table.
To ensure this is mounted at boot time, add the following to /etc/fstab:
fdescfs /dev/fd fdescfs rw 0 0
To ensure you have the latest versions of the spam blacklists, you can refresh
them once per hour with this line in /etc/crontab:
48 * * * * /usr/local/sbin/spamd-setup
The spamd-setup utility adds blacklists by adding addresses to the pf table
<spamd> according to the instructions in /usr/local/etc/spamd.conf.
To distribute the load a bit, and not have everyone hitting the servers at the
same time (e.g. 48 minutes past the hour, or at the top of the hour), change
48 to whatever minute is is when you enter the crontab entry.
You’ll need a copy of spamd.conf:
cp /usr/local/etc/spamd.conf.sample /usr/local/etc/spamd.conf
You may wish to amend spamd.conf according to your needs. Personally, I changed
the following:
all:\ :spamhaus:china:korea:
to:
all:\ :spamhaus:spews1
I also added this to /etc/syslog.conf so I could see the log from spamd:
!spamd daemon.err;daemon.warn;daemon.info /var/log/spamd
I also issued this command to create the file:
touch /var/log/spamd
Remember to HUP syslogd so it reads your changes and takes appropriate action:
kill -HUP `cat /var/run/syslog.pid`
Although your log file will be empty at this point, here are a few entries that appeared
after it had been running for a while.
$ tail /var/log/spamd Nov 8 00:30:15 nyi spamd[27528]: 212.12.70.131: connected (1/0) Nov 8 00:30:15 nyi spamd[27528]: 212.12.70.131: disconnected after 0 seconds. Nov 8 00:37:31 nyi spamd[27528]: 210.4.36.220: connected (1/0) Nov 8 00:37:34 nyi spamd[27528]: (GREY) 210.4.36.220: <deborahmckenzie_kg@browningdirect.example.com> -> >papers@bsdcan.example.org> Nov 8 00:37:34 nyi spamd[27528]: 210.4.36.220: disconnected after 3 seconds. Nov 8 00:37:38 nyi spamd[27528]: 210.4.36.220: connected (1/0) Nov 8 00:37:40 nyi spamd[27528]: (GREY) 210.4.36.220: >deborahsee@broadwayrealestate.example.com> -> <papers@bsdcan.example.org> Nov 8 00:37:40 nyi spamd[27528]: 210.4.36.220: disconnected after 2 seconds. Nov 8 00:45:16 nyi spamd[27528]: 69.133.112.184: connected (1/0) Nov 8 00:45:16 nyi spamd[27528]: 69.133.112.184: disconnected after 0 seconds.
No, those aren’t the real email addresses from my logs, but they are close.
Known good mailers that have trouble with greylisting
There are some problems with greylisting. Have a read of the whitelisting
section at greylisting.org. I have
taken their whitelist and added it to my whitelist (/usr/local/etc/spamd-mywhite).
Also in that file are my own mailservers and any special places which are immune
from any spamd intervention.
Directing things around the tarpit
Here are the rules I added to /etc/pf.rules:
- table <spamd> persist
- table <spamd-white> persist
- table <spamd-mywhite> persist file "/usr/local/etc/spamd-mywhite"
- scrub in all
- # redirect to spamd
- rdr pass inet proto tcp from <spamd-white> to $external_addr port \ smtp -> 127.0.0.1 port smtp
- rdr pass inet proto tcp from <spamd> to $external_addr port \ smtp -> 127.0.0.1 port spamd
- rdr pass inet proto tcp from !<spamd-mywhite> to $external_addr port \ smtp -> 127.0.0.1 port spamd
- # mail!
- pass in log inet proto tcp from any to $external_addr port smtp flags S/SA \ synproxy state
- pass out log inet proto tcp from $external_addr to any port smtp flags S/SA \ synproxy state
Notes for the above:
- 1 – declare the spamd table. Everyone in this table is redirected to spamd.
The contents of this table is maintained by spamd-setup, which is run from a
cronjob. - 2 – declare the whitelist table of locations that have successfully negotiated
spamd’s greylisting efforts. This table is maintained by spamlogd. - 3 – declare a table of locations that should not be subjected to greylisting.
I maintain this table manually. - 8 – Everyone in the whitelist goes straight to the mail server
- 9 – Everyone on the greylist goes straight to spamd
- 10 – Everyone not on my whitelist gets to talk to spamd
- 13-14 – These lines will feed data to pflog, which spamlogd will monitor
and use to update the spamd table.
NOTE: The use of pass on the RDR rules is significant. The following is borrowed
from the OpenBSD PF FAQ
NAT and Packet Filtering
NOTE: Translated packets must still pass
through the filter engine and will be blocked or passed based on the
filter rules that have been defined.
The only exception to this rule is when the pass keyword
is used within the nat rule.
This will cause the NATed packets to pass right through the filtering
engine.Also be aware that since translation occurs before filtering, the
filter engine will see the translated packet with the translated
IP address and port as outlined in How NAT Works.
In short, if you’re on a whitelist (either spamd’s whitelist or my whitelist),
you go straight to the mail server. Everyone else, goes to spamd.
On your first visit to spamd, you are asked to come back later. If you do, then
you’re asked to try again, and you are added to the whitelist.
By the way, after making changes to /usr/local/etc/spamd-mywhite, to tell PF
to take notice of those changes, I issue this command:
$ pfctl -t spamd-mywhite -T replace -f /usr/local/etc/spamd-mywhite No ALTQ support in kernel ALTQ related functions disabled 24 addresses added. 39 addresses deleted.
Starting spamd
To start spamd manually, issue this command:
/usr/local/etc/rc.d/pfspamd start
The first time you run spamd, it may take 10 or 20 seconds for it to come back
to the command line. Be patient.
If you get the following error message:
# /usr/local/etc/rc.d/pfspamd start Starting pfspamd. spamd-setup: Can't find "all" in spamd config: No such file or directory
then you probably forget to create /usr/local/etc/spamd.conf (see above) or
you removed the all
section from it.
You will also want to start pflogd, so that the logging works:
/etc/rc.d/pflog start
After successfully starting spamd, you should see stuff like this:
# ps auwx | grep pf nobody 94067 ?? Ss 11:20AM 0:00.03 spamd: (pf <spamd-white> update) (spamd) root 94282 ?? Is 11:31AM 0:00.00 pflogd: [priv] (pflogd) _pflogd 94286 ?? S 11:31AM 0:00.00 pflogd: [running] -s 116 -f /var/log/pflog (pflogd)
I have removed some of the columns from the above display to make it fit better on the page.
On a side note, I’d like to see spamd running as something other than nobody.
Perhaps I’ll work on that later.
spamd-setup maintains the <spamd> table shown on line 1 of the PF
ruleset found in a later section of this article. To view the contents of
this table, issue this command:
pfctl -t spamd -T show
Getting things from the greylist into the whitelist
You don’t have to worry about moving items from the greylist to the whitelist.
spamlogd will take care of that for you. If you’re setting this up for the
first, time, you can get spamlogd running with this command:
/usr/local/libexec/spamlogd
To ensure spamlogd starts at boot time, include this in /etc/rc.conf:
pfspamlogd_enable="YES"
spamlogd updates the spamd database (/var/db/spamd). When it sees a successful
connection spamd, in turn, uses
this database to decide whether someone is on the whitelist or greylist.
In order to provide spamlogd with the information it needs, you must log
your mail server activity. See lines 13 & 14 above.
Read all the details in
man spamlogd.
If spamlogd does not start, it is probably because pflogd is not running. See
above for starting pflogd. This is what spamlogd looks like when it is running:
# ps auwx | grep spamlogd root 94345 ?? Ss 11:36AM 0:00.00 /usr/local/libexec/spamlogd root 94349 p2 S+ 11:36AM 0:00.00 grep spamlogd
A sample greylisting
In this section, I will send a message from a non-whitelisted server and
demonstrate how the server moves from the greylist to the whitelist. I will
be sending from dan@zip.example.org to dan@nyi.example.org. For your information,
zip is running sendmail, and nyi is running postfix. In both cases, that
is completely irrelevant to greylisting.
Here is an extract from the sending mailserver. I guess I should point out that
this server is in New Zealand and the one I’m sending to is in New York.
Nov 9 06:30:06 zip sm-mta[59825]: kA8HThYO059822: to=<dan@nyi.example.org>, ctladdr=<dan@zip.example.org> (1001/1001), delay=00:00:20, xdelay=00:00:20, mailer=esmtp, pri=30391, relay=nyi.example.org. [64.147.113.42], dsn=4.3.0, stat=Deferred: 451 Temporary failure, please try again later.
A new host, zip.example.org, previously unknown to nyi.example.org, attempted
to send email. spamd on nyi correctly asked zip to try again. Checking the mail
queue on zip, you should see something like this:
$ mailq /var/spool/mqueue (1 request) -----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient----------- kA8HThYO059822 34 Thu Nov 9 06:29 <dan@zip.example.org> (Deferred: 451 Temporary failure, please try again later.) <dan@nyi.example.org> Total requests: 1
Looking at the logs on nyi, I see this in /var/log/spamd:
Nov 8 12:29:58 nyi spamd[27528]: 203.118.144.46: connected (1/0)
Nov 8 12:29:59 nyi spamd[27528]: (GREY) 203.118.144.46: <dan@zip.example.org> -> <dan@nyi.example.org>
Nov 8 12:29:59 nyi spamd[27528]: 203.118.144.46: disconnected after 1 seconds.
Furthermore, you can see that zip is greylisted by issuing this command:
$ spamdb | grep nz
GREY|203.118.144.46|<dan@zip.example.org>|<dan@nyi.example.org>|1163006999|1163021399|1163021399|1|0
I waited. Shortly thereafter, zip tries again, and again spamd asks it to try
again. Here is the log entry from zip, the sending mailserver:
Nov 9 06:43:02 zip sm-mta[59893]: kA8HThYO059822: to=<dan@nyi.example.org>, ctladdr=<dan@zip.example.org> (1001/1001), delay=00:13:16, xdelay=00:00:05, mailer=esmtp, pri=120391, relay=nyi.example.org. [64.147.113.42], dsn=4.3.0, stat=Deferred: 451 Temporary failure, please try again later.
Checking on nyi, I looked in the spamd database again:
$ spamdb | grep 203.118.144.46
GREY|203.118.144.46|<dan@zip.example.org>|<dan@nyi.example.org>|1163006999|1163021399|1163021399|2|0
There it is, clear as day. The entry has been greylisted. The three numeric
fields indicate timestamps related to this host. The 2 means the host
has attempted delivery twice. The zero means the host has not yet delivered
any mail.
Why was the second attempt not allowed? spamd has three time parameters
related to greylisting. See the man page for better definitions. The values
shown are the defaults.
- passtime – If after this time period, spamlogd sees a retried delivery,
the server will be moved to the whitelist. (25 minutes) - greyexp – entries on the greylist will be removed if there have been no
retries within this period. (4 hours) - whiteexp – Entries on the whitelist are removed if there has been no
mail activity in this time period (36 days).
The default passtime value (see man spamd)
is 25 minutes. A host will remain greylisted for at least 25 minutes before it
can be moved to the whitelist. What will move it to the whitelist? A retry after
passtime minutes. This means that delivery must be attempted three times, and
it will succeed on the third try, if it it after the passtime period and before
the greyexp period terminates. By default, the sending mail server will be greylisted
for 25 minutes, and then has until 4 hours after the first delivery attempt to
try again. After the greylisting period expires, the sending host must go
through the greylisting process again.
So now I waited, for another attempt. And there it was:
Nov 9 07:12:59 zip sm-mta[60044]: kA8HThYO059822: to=<dan@nyi.example.org>, ctladdr=<dan@zip.example.org> (1001/1001), delay=00:43:13, xdelay=00:00:02, mailer=esmtp, pri=210391, relay=nyi.example.org. [64.147.113.42], dsn=4.3.0, stat=Deferred: 451 Temporary failure, please try again later.
Remember how I said spamlogd will monitor the mail logs and move entries to the
white list? Look here on nyi:
$ spamdb | grep 203.118.144.46 WHITE|203.118.144.46|||1163006999|1163009572|1166119999|3|0
There you go, the entry has been whitelisted. Three delivery attempts have
been made, and none have succeeded. Actually, as I type this, the mail
is still sitting on zip waiting to attempt another delivery. This time,
PF will redirect the incoming connection directly to my mail server, and not
to spamd.
Coincident with the above was this log entry:
*** /var/log/debug.log *** Nov 8 13:13:19 nyi spamd[27526]: whitelisting 203.118.144.46 in /var/db/spamd
That is spamlogd reporting that it has whitelisted the client. On the next
delivery attempt, the message should go straight through. And here it is:
Nov 9 07:43:06 zip sm-mta[60179]: kA8HThYO059822: to=<dan@nyi.example.org>, ctladdr=<dan@zip.example.org> (1001/1001), delay=01:13:20, xdelay=00:00:09, mailer=esmtp, pri=300391, relay=nyi.example.org. [64.147.113.42], dsn=2.0.0, stat=Sent (Ok: queued as 661DC50849)
And here it is being received on nyi:
Nov 8 13:42:56 nyi postfix/smtpd[2847]: connect from zip.example.org[203.118.144.46]
Nov 8 13:42:58 nyi postfix/smtpd[2847]: 661DC50849: client=zip.example.org[203.118.144.46]
Nov 8 13:42:59 nyi postfix/cleanup[2873]: 661DC50849: message-id=<200611081729.kA8HTfX6059821@zip.example.org>
Nov 8 13:42:59 nyi postfix/qmgr[47862]: 661DC50849: from=<dan@zip.example.org>, size=852, nrcpt=1 (queue active)
Nov 8 13:42:59 nyi postfix/local[2874]: 661DC50849: to=<dan@nyi.example.org>, relay=local, delay=2.3, delays=2.3/0.01/0/0.01, dsn=2.0.0, status=sent (delivered to command: exec /usr/local/bin/procmail -t -a ${EXTENSION})
Nov 8 13:42:59 nyi postfix/qmgr[47862]: 661DC50849: removed
The host is now in the spamd database as this:
$ spamdb | grep 203.118.144.46 WHITE|203.118.144.46|||1163006999|1163009572|1166121773|3|1
203.118.144.46 will remain on the whitelist until 36 days of no sent email.
While on the whitelist, it will not be subject to greylisting. If if falls
off the whitelist, it will go through the above greylisting process again.
Problems I encountered
Yes, I had a problem. The <spamd-white> table was always empty.
Even after spamlogd moved something from the greylist to the whitelist.
This thread
in the FreeBSD PF mailing list found me the missing piece:
mount -t fdescfs fdescfs /dev/fd
Now the whitelist table has some entries!
# pfctl -t spamd-white -T show No ALTQ support in kernel ALTQ related functions disabled 12.152.184.25 66.35.250.206 205.150.199.217 216.136.204.119 #
My thanks to LI Xin.
Now, an even bigger problem. Despite having the whitelist updated, my email
still isn’t getting through. Whitelisted clients were still being subjected
to greylisting. delo found the answer. My rules were wrong. I had this:
- rdr pass inet proto tcp from <spamd-mywhite> to $external_addr port \ smtp -> 127.0.0.1 port smtp
- rdr pass inet proto tcp from <spamd> to $external_addr port \ smtp -> 127.0.0.1 port spamd
- rdr pass inet proto tcp from !<spamd-mywhite> to $external_addr port \ smtp -> 127.0.0.1 port spamd
See the problem? Line 1. That table name is wrong. It should be
<spamd-white>, the table maintained by spamlogd. DOH! When I found this
problem, I corrected /etc/pf.rules and updated PF by issuing this command:
pfctl -f /etc/pf.rules
Later on, I found out about this option:
-N Load only the NAT rules present in the rule file. Other rules and options are ignored.
Once I fixed the NAT rules, I went to m21 and tried to connect. I got straight
through to the real smtp server:
dan@m21:~$ telnet nyi 25 Trying 64.147.113.42... Connected to nyi.example.org. Escape character is '^]'. 220 nyi.example.org ESMTP Postfix QUIT 221 2.0.0 Bye Connection closed by foreign host. dan@m21:~$
Good, that proves the whitelisting is working. Then I flushed the Postfix mail
queue, and the mail message went straight through.
Yes, I missed this entirely during the port install:
$ cd /usr/ports/mail/spamd $ less pkg-message ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ In order to use spamd greylisting feature you have to have a mounted fdescfs(5) at /dev/fd. This is done by adding: fdescfs /dev/fd fdescfs rw 0 0 to /etc/fstab. You may need either a customized kernel, or kldload the fdescfs kernel module. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ $
What is in my spambd right now?
$ spamdb | grep GREY
GREY|12.199.121.98|<abfaf@cardinalconst.example.com>|<sponsorship@bsdcan.example.org>|1163008607|1163023007|1163023007|1|0
GREY|12.199.121.98|<bfddcgfaceccbe@carltonabbott.example.com>|<sponsorship@bsdcan.example.org>|1163008652|1163023052|1163023052|1|0
GREY|12.199.121.98|<daacgdcedacg@careerpointgroup.example.com>|<sponsorship@bsdcan.example.org>|1163008622|1163023022|1163023022|1|0
GREY|12.199.121.98|<eegadda@carrierescrete.example.com>|<sponsorship@bsdcan.example.org>|1163008592|1163022992|1163022992|1|0
GREY|12.199.121.98|<gffafgfd@cascadecont.example.com>|<sponsorship@bsdcan.example.org>|1163008636|1163023036|1163023036|1|0
GREY|199.227.43.178|<wow_deb48@sharkteethrus.example.com>|<sponsorship@bsdcan.example.org>|1163002782|1163017182|1163017182|1|0
GREY|201.216.157.1|<BrunoYang@rotes-teufelchen.de>|<papers@bsdcan.example.org>|1163005081|1163019481|1163019481|1|0
GREY|201.216.157.1|<OctavioDickey@rpcredit.ie>|<papers@bsdcan.example.org>|1163005080|1163019480|1163019480|1|0
GREY|202.27.236.89|<>|<info@bsdcan.example.org>|1163010937|1163025337|1163025337|1|0
GREY|213.98.26.251|<noreply@freebsddiary.example.org>|<majordomo@freebsddiary.example.org>|1163011838|1163026238|1163026238|1|0
GREY|41.241.113.223|<febo@geol.lsu.edu>|<majordomo@freebsddiary.example.org>|1163011853|1163026253|1163026253|1|0
GREY|58.8.10.140|<ppppppp@hotmail.example.com>|<dan@langille.example.org>|1163001747|1163016147|1163016147|2|0
GREY|58.8.99.137|<thaiwork_job@yahoo.example.com>|<papers@bsdcan.example.org>|1163002285|1163016685|1163016685|1|0
GREY|62.215.92.138|<training@intech-online.example.com>|<dan@langille.example.org>|1163010826|1163025226|1163025226|1|0
GREY|62.45.20.12|<deboraholcomb_hu@calcoastrepiping.example.com>|<activities@bsdcan.example.org>|1163000304|1163014704|1163014704|1|0
GREY|62.45.20.12|<deborahtaylor235@campuscrossroads.example.com>|<activities@bsdcan.example.org>|1163000292|1163014692|1163014692|1|0
GREY|75.89.28.189|<john@pistonheads.biz>|<payment@bsdcan.example.org>|1162997706|1163012106|1163012106|1|0
GREY|76.184.184.115|<stephen@quasarman.biz>|<payment@bsdcan.example.org>|1163008485|1163022885|1163022885|1|0
GREY|80.35.70.14|<mingshengw@postcardsally.example.com>|<payment@bsdcan.example.org>|1163010212|1163024612|1163024612|1|0
GREY|80.98.245.220|<Antelmi@care-mail.example.com>|<papers@bsdcan.example.org>|1163007068|1163021468|1163021468|1|0
GREY|84.227.161.190|<rooster@tuttoocchiali.example.com>|<keys@bsdcan.example.org>|1163001318|1163015718|1163015718|1|0
GREY|84.245.217.46|<work96@tel.fer.hr>|<majordomo@freebsddiary.example.org>|1163005846|1163020246|1163020246|1|0
GREY|84.60.218.15|<sensirox.example.com@theloglog.example.com>|<activities@bsdcan.example.org>|1163002484|1163016884|1163016884|1|0
GREY|85.98.190.1|<deborahschlumpf@calabreselaw.example.com>|<sponsorship@bsdcan.example.org>|1163009003|1163023403|1163023403|1|0
GREY|85.98.190.1|<deborapadinha@canaltai.example.com>|<sponsorship@bsdcan.example.org>|1163009013|1163023413|1163023413|1|0
GREY|91.76.45.94|<h-dudaz@usa.net>|<majordomo@freebsddiary.example.org>|1163003291|1163017691|1163017691|1|0
Yes, the domain names have been slightly obscured, but you should be able to see
who is sending to what. For the record, the MX server in question is not an
MX for langille.org or freebsddiary.org… but that’s not stopping the spammers
from trying. At present, only bsdcan.org uses this greylisting server as an MX.
I’m about to add more domains to it and implement greylisting on my other servers.
As I type this additional note on 24 November, about 3 weeks after the above, here are the stats of each of my
three mail servers:
- nyi
$ spamdb | grep -c GREY 101 $ spamdb | grep -c WHITE 4462
- havoc
$ spamdb | grep -c GREY 256 $ spamdb | grep -c WHITE 2404
- supernews
$ spamdb | grep -c GREY 30 $ spamdb | grep -c WHITE 37
It is interesting to see that one machine has whitelisted nearly 4500 servers in about 9 days…
Greytrapping
I’m sure all of this sounds great. It can be better. Greytrapping is one step further than greylisting.
No doubt, you have an email address that is no longer used, but you still get email sent to it. It’s
probably been on spamming lists for years. If someone is sending email to that address, it’s bound
to be spam. What you can do is add that address to spamdb as a spamtrap address. See man spamdb for
details. For example, if you want to designate anyone sending to yourname@example.org, then you can
issue this command:
spamdb -T -a "<yourname@example.org>"
I have a list of 24592 such email addresses. Why? Well, they aren’t really addresses. They are
Message-ID: values from FreshPorts. Background: FreshPorts didn’t
always store Message-ID:. When that attribute was added, I need to come up with a value for the existing
commits stored in the database. Unfortunately, I select something like fp1.12345@example.org (s/example/FreshPorts).
Spammers grabbed all those addresses, and I started to see huge spam attempts. All bounced of course,
because they were not valid addresses. I have since changed those Message-ID:s to @dev.null.example.org
(s/example/FreshPorts). But the spammers continue.
So how do I get the email addresses into spamdb? They are all in a file named greytrap.
This command loads them. It takes a few minutes to complete.
cat greytrap | xargs -n1 spamdb -T -a
That’s all there is to it…
Greyscanning
With newer versions of spamd (not available in the FreeBSD Ports tree at the time of writing),
you can take advantage of the greylisting period to scan your logs and take appropriate action.
The greyscanner script will scan the spamdb output and look for patterns and blacklist that IP address
for 24 hours. If it’s not spam, it will come through later. If it is spam, well, you’ve delayed it.
This script can validate address, check for an MX or A record for the source address, etc.
Look here for details: http://www.ualberta.ca/~beck/nycbug06/scripts/
Things to think about
Greylisting can delay mail. Greylisting can block mail, but only if you continuously redirect the connection to the tarpit.
However, it does greatly reduce the amount of incoming spam. I have no comparative statistics to show you. All I know is I like
it and it reduces the amount of crap in my mail box. 🙂
A link in the first paragraph is broken:
Bob Beck’s talk on spamd
Should be:
<http://www.nycbsdcon.org/2006/speakers#Beck>
Thank you for this article, looking into spamd has been on my todo-list for some time, and this article and this article made it a whole lot easier.
Fixed.
Thank you.
background: The NYCBUG website moved stuff around… they archived stuff into the 2006 directory.
Glad you got pf running.
cheers.
—
The Man Behind The Curtain
This is quite useful. In case anyone is wondering, The package pfspamd is now known as obspamd.
[%sig%]