Everything got owned
NOTE: I wrote this article on 23 November, as things were happening, and as I worked on the problem.
This morning was not a good start. First, all my IRC connections had died. Checking with
Netsaint, all my boxes were offline. Great! My
gateway is having problems. I ssh’d to my gateway. ifconfig showed it had a
public IP address. I could ping that IP address. netstat -nr showed I had a
gateway. I could ping that. I could also ping google.ca. OK, the gateway
works. Next, I checked my browser, yes, it could get to Google. OK, so what’s
wrong?
I could not ssh to any of my boxes in New Zealand, Vancouver, Ottawa, or San
Jose. They were all affected…
I got onto IRC and started chatting about this. Suggestions where made that I got
owned. I doubted it. Netsaint showed all the problems started at the same time
(give or take a few minutes). I tried ssh’ing to another NZ-based box,
and I got in. OK, so all is not bad. That was one of the boxes listed as a
problem by Netsaint.
OK, what’s up?
Blame Canada
Actually, don’t blame Canada. Blame DNS. All of these problems are explained
by a DNS change. The public IP address on my gateway changed. Lovely!
This explains everything. All the problems noted by Netsaint were related to
Netsaint client calls… Ping and http were fine. I missed that part.
The firewall rules on the boxes restrict incoming connections by IP address.
That explains why Netsaint was complaining. Here is what it said for the
NYI box:
Notice that ping and http are fine. These services are checked directly by Netsaint.
The other services are checked by connecting to a port on the remote box and
talking to netsaint_statd. However, I have configured netsaint_statd to speak
only to a specific IP address; one that I no longer have access to.
Dynamic DNS
It is because I have a dynamic IP address that I have been using the
dns/noip port since I started
with Rogers. Why did I go with Rogers?
I called them on June 30. They were here and had installed my connection within
18 hours. In fact, they showed up on a statutory holiday, and took less then
15 minutes to get me hooked up. I loved that. I tried to get DSL, but they
said it would take more than a week. And I’d have to use PPPoE. With cable,
I have direct ethernet
Despite having a dynamic IP address, this is the first time it has changed. I was
never sure if noip would work or not. Well, it worked. Invisibly. Witness:
$ host bast.unixathome.org bast.unixathome.org is an alias for dlangille.no-ip.org. dlangille.no-ip.org has address 74.104.199.163 bast.unixathome.org is an alias for dlangille.no-ip.org. bast.unixathome.org is an alias for dlangille.no-ip.org.
Yep, that’s right. All my hostnames that point to my servers at home use that
hostname:
$ host beta.freshports.org beta.freshports.org is a nickname for dlangille.no-ip.org dlangille.no-ip.org has address 74.104.199.163
That’s good. I don’t have to run around and update all my DNS. That investment
has paid off.
Gateway firewall rules
You see that FreshPorts BETA has the
right IP address. That’s good. But nobody could get to the websites. Why?
My firewall rules. They have the external IP address hard-coded. That would
be OK if that was static, but it’s not. Here’s what I had:
external_addr="70.26.229.230"
In the ruleset, you’ll see this:
rdr on $ext_if proto tcp from any to $external_addr port http -> $webserver
That’s a problem.
How do you avoid hardcoding an IP address that changes? pf can handle that.
mux, brix, and dnns all pointed out a couple of solutions. Namely:
- self – From http://www.openbsd.org/faq/pf/tables.html:
Specifying Addresses
In addition to being specified by IP address, hosts may also be specified by
their hostname. When the hostname is resolved to an IP address, all resulting
IPv4 and IPv6 addresses are placed into the table. IP addresses can also be
entered into a table by specifying a valid interface name or the self keyword.
The table will then contain all IP addresses assigned to that interface or to
the machine (including loopback addresses), respectively.
OK, that’s useful. Using a hostname is good. But when the IP address changes,
the ruleset is not updated, so it is not quite what I need. - ($ext_if) – From http://www.openbsd.org/faq/pf/filter.html:
The name of a network interface or group in parentheses ( ).
This tells PF to update the rule if the IP address(es) on the named interface
change. This is useful on an interface that gets its IP address via DHCP or
dial-up as the ruleset doesn’t have to be reloaded each time the address changes.
Yes, that is just what I need for my home gateway!
Changing the rule set was easy. I removed the
external_addr=”70.26.229.230″ entry. I then replaced all instances of
$external_addr with ($ext_if). To test those rule changes:
# pfctl -n -f /etc/pf.rules #
And then to invoke those new rules:
# pfctl -f /etc/pf.rules #
With those changes, http://beta.freshports.org/
was back online. And so were all the other websites I have at home. Similarly,
my mail started flowing in as well…. A good start. Now, let’s go look at those servers.
Back to the servers…
Now that I have the beta websites working and mail flowing, now I need to gain
access to my servers. All the servers restrict incoming ssh to specific IP
addresses. One of them has a serial console connection. I’ve been able to ssh
to each box and update the firewall rules and reconfigure netsaint_statd.
The changes to the firewall rules consisted of updating one line in the PF configuration
file. I had used a macro to define the IP address of my home box, so updating the rules
was quite painless, once I could get to that box.
I had to change this line in netsaint_statd:
my @allowed_hosts = ("74.104.199.163");
I had to kill and restart that netsaint_statd. It is not set up to take a HUP
and /usr/local/etc/rc.d/netsaint_statd.sh does not take a stop or restart argument.
I also updated /etc/hosts and /etc/hosts.allow
When I visited a few other hosts, I found a problem. Can you see it?
my @allowed_hosts; # = ("10.55.0.18");
For starters, that’s the wrong IP address. More importantly, there is a comment
in there.. OUCH. I corrected that too, on a number of hosts.
Other notes
Just after I got my mail flowing, and websites back online, I started checking
my email. In my logs, I found this:
Nov 23 22:31:59 bast dhclient: New IP Address (fxp0): 74.104.199.163 Nov 23 22:31:59 bast dhclient: New Subnet Mask (fxp0): 255.255.255.128 Nov 23 22:31:59 bast dhclient: New Broadcast Address (fxp0): 255.255.255.255 Nov 23 22:31:59 bast dhclient: New Routers (fxp0): 74.104.199.129 Nov 23 22:32:20 bast noip2[76688]: execl /usr/home/dan/bin/ip-updated.sh failed (Permission denied)
Well! Not that it would have helped me, because I didn’t check my email first thing,
but there’s my first, yet unfound until now, clue about the problem.
I fixed the problem listed above:
# ls -l /usr/home/dan/bin/ip-updated.sh -rw-r--r-- 1 dan dan 76 Jul 1 2005 /usr/home/dan/bin/ip-updated.sh # chmod +x /usr/home/dan/bin/ip-updated.sh # cat /usr/home/dan/bin/ip-updated.sh !#/bin/sh echo 'IP address has been updated' | mail dan@xeon.example.org #
NOTE: 29 November 2006: As I type this, I have no outside connectivity. I suspect
a routing problem outside my gateway. I checked the logs, and found what I thought
was another IP address change. I was wrong. I was looking at the incident from 23
Nov. However, all is not lost. I did notice that the above permission problem
was caused by a typo, not a permission issue.
!#/bin/sh should be #!/bin/sh. I just called my ISP. There is
a problem and they’re working on it.
Where is that script called from? That took me a few minutes to track down. I
looked at /usr/local/share/doc/noip/README.FIRST:
# /usr/local/bin/noip2 -S
1 noip2 process active.
Process 1030, started as /noip2
Using configuration from /usr/local/etc/no-ip2.conf
Last IP Address set 74.104.199.163
Account someone@example.com
configured for:
host dlangille.no-ip.org
Executing /usr/home/dan/bin/ip-updated.sh upon successful update.
Address check every 1 minute, directly connected via /dev/fxp0.
NOTE: email addresses in this section have been obscured.
For what it is worth, my friend Eric Rosenquist also had an IP address change
last night. At 20:33:11. He is also with Rogers.
Trailing notes
What did I learn?
- I improved my firewall rules so they did not refer to IP addresses that can change.
- I corrected problems with my netsaint_statd configuration on a few clients.
- I fixed my no-ip.org configuration so that email notifications work.
I restrict ssh logins to specify IP addresses. That is why I could not ssh in from home. I also allow
ssh from a few servers that do not restrict incoming ssh. Coincidentally, one of those servers was offline
at the time.
In short, it wasn’t a disaster. Nothing was owned. If IP address changes happened more regularly, then I think
I would have known sooner what the problem was. Let’s see what happens next time. I’m better prepared now.
One last note: As I went to publish this article, I realized another
configuration file that needed to be updated: stunnel.conf. I use stunnel
to create a secure tunnel between my cvs server and my webserver. This
enables cvsup to work in private and without exposing the service to snooping
eyes. For what it’s worth, the configuration file now uses a hostname,
not an IP address. That means I will just need to restart the service in
future.