Nov 272006
Everything got owned
NOTE: I wrote this article on 23 November, as things were happening, and as I worked on the problem. This morning was not a good start. First, all my IRC connections had died. Checking with Netsaint, all my boxes were offline. Great! My gateway is having problems. I ssh’d to my gateway. ifconfig showed it had a public IP address. I could ping that IP address. netstat -nr showed I had a gateway. I could ping that. I could also ping google.ca. OK, the gateway works. Next, I checked my browser, yes, it could get to Google. OK, so what’s wrong? I could not ssh to any of my boxes in New Zealand, Vancouver, Ottawa, or San Jose. They were all affected… I got onto IRC and started chatting about this. Suggestions where made that I got owned. I doubted it. Netsaint showed all the problems started at the same time (give or take a few minutes). I tried ssh’ing to another NZ-based box, and I got in. OK, so all is not bad. That was one of the boxes listed as a problem by Netsaint. OK, what’s up?Blame Canada
Actually, don’t blame Canada. Blame DNS. All of these problems are explained by a DNS change. The public IP address on my gateway changed. Lovely! This explains everything. All the problems noted by Netsaint were related to Netsaint client calls… Ping and http were fine. I missed that part. The firewall rules on the boxes restrict incoming connections by IP address. That explains why Netsaint was complaining. Here is what it said for the NYI box:
Dynamic DNS
It is because I have a dynamic IP address that I have been using the dns/noip port since I started with Rogers. Why did I go with Rogers? I called them on June 30. They were here and had installed my connection within 18 hours. In fact, they showed up on a statutory holiday, and took less then 15 minutes to get me hooked up. I loved that. I tried to get DSL, but they said it would take more than a week. And I’d have to use PPPoE. With cable, I have direct ethernet Despite having a dynamic IP address, this is the first time it has changed. I was never sure if noip would work or not. Well, it worked. Invisibly. Witness:Yep, that’s right. All my hostnames that point to my servers at home use that hostname:$ host bast.unixathome.org bast.unixathome.org is an alias for dlangille.no-ip.org. dlangille.no-ip.org has address 74.104.199.163 bast.unixathome.org is an alias for dlangille.no-ip.org. bast.unixathome.org is an alias for dlangille.no-ip.org.
That’s good. I don’t have to run around and update all my DNS. That investment has paid off.$ host beta.freshports.org beta.freshports.org is a nickname for dlangille.no-ip.org dlangille.no-ip.org has address 74.104.199.163
Gateway firewall rules
You see that FreshPorts BETA has the right IP address. That’s good. But nobody could get to the websites. Why? My firewall rules. They have the external IP address hard-coded. That would be OK if that was static, but it’s not. Here’s what I had:In the ruleset, you’ll see this:external_addr="70.26.229.230"
That’s a problem. How do you avoid hardcoding an IP address that changes? pf can handle that. mux, brix, and dnns all pointed out a couple of solutions. Namely:rdr on $ext_if proto tcp from any to $external_addr port http -> $webserver
- self – From http://www.openbsd.org/faq/pf/tables.html:
Specifying Addresses In addition to being specified by IP address, hosts may also be specified by their hostname. When the hostname is resolved to an IP address, all resulting IPv4 and IPv6 addresses are placed into the table. IP addresses can also be entered into a table by specifying a valid interface name or the self keyword. The table will then contain all IP addresses assigned to that interface or to the machine (including loopback addresses), respectively.
OK, that’s useful. Using a hostname is good. But when the IP address changes, the ruleset is not updated, so it is not quite what I need. - ($ext_if) – From http://www.openbsd.org/faq/pf/filter.html:
The name of a network interface or group in parentheses ( ). This tells PF to update the rule if the IP address(es) on the named interface change. This is useful on an interface that gets its IP address via DHCP or dial-up as the ruleset doesn’t have to be reloaded each time the address changes.
Yes, that is just what I need for my home gateway!
And then to invoke those new rules:# pfctl -n -f /etc/pf.rules #
With those changes, http://beta.freshports.org/ was back online. And so were all the other websites I have at home. Similarly, my mail started flowing in as well…. A good start. Now, let’s go look at those servers.# pfctl -f /etc/pf.rules #
Back to the servers…
Now that I have the beta websites working and mail flowing, now I need to gain access to my servers. All the servers restrict incoming ssh to specific IP addresses. One of them has a serial console connection. I’ve been able to ssh to each box and update the firewall rules and reconfigure netsaint_statd. The changes to the firewall rules consisted of updating one line in the PF configuration file. I had used a macro to define the IP address of my home box, so updating the rules was quite painless, once I could get to that box. I had to change this line in netsaint_statd:
my @allowed_hosts = ("74.104.199.163");
my @allowed_hosts; # = ("10.55.0.18");
Other notes
Just after I got my mail flowing, and websites back online, I started checking my email. In my logs, I found this:Well! Not that it would have helped me, because I didn’t check my email first thing, but there’s my first, yet unfound until now, clue about the problem. I fixed the problem listed above:Nov 23 22:31:59 bast dhclient: New IP Address (fxp0): 74.104.199.163 Nov 23 22:31:59 bast dhclient: New Subnet Mask (fxp0): 255.255.255.128 Nov 23 22:31:59 bast dhclient: New Broadcast Address (fxp0): 255.255.255.255 Nov 23 22:31:59 bast dhclient: New Routers (fxp0): 74.104.199.129 Nov 23 22:32:20 bast noip2[76688]: execl /usr/home/dan/bin/ip-updated.sh failed (Permission denied)
NOTE: 29 November 2006: As I type this, I have no outside connectivity. I suspect a routing problem outside my gateway. I checked the logs, and found what I thought was another IP address change. I was wrong. I was looking at the incident from 23 Nov. However, all is not lost. I did notice that the above permission problem was caused by a typo, not a permission issue. !#/bin/sh should be #!/bin/sh. I just called my ISP. There is a problem and they’re working on it. Where is that script called from? That took me a few minutes to track down. I looked at /usr/local/share/doc/noip/README.FIRST:# ls -l /usr/home/dan/bin/ip-updated.sh -rw-r--r-- 1 dan dan 76 Jul 1 2005 /usr/home/dan/bin/ip-updated.sh # chmod +x /usr/home/dan/bin/ip-updated.sh # cat /usr/home/dan/bin/ip-updated.sh !#/bin/sh echo 'IP address has been updated' | mail dan@xeon.example.org #
# /usr/local/bin/noip2 -S
1 noip2 process active.
Process 1030, started as /noip2
Using configuration from /usr/local/etc/no-ip2.conf
Last IP Address set 74.104.199.163
Account someone@example.com
configured for:
host dlangille.no-ip.org
Executing /usr/home/dan/bin/ip-updated.sh upon successful update.
Address check every 1 minute, directly connected via /dev/fxp0.
Trailing notes
What did I learn?- I improved my firewall rules so they did not refer to IP addresses that can change.
- I corrected problems with my netsaint_statd configuration on a few clients.
- I fixed my no-ip.org configuration so that email notifications work.