Managing an IP address change – part 2
It was almost a month ago that I started changing the IP address of The FreeBSD Diary web server. When I first mentioned to someone that I was going to write an article about this process, they wondered: why, all it takes is an ifconfig, what is there to write about? I guess they didn’t understand the whole story. Yes, if you’re changing the IP address of a box which is not providing services, then no, it’s not much of a big deal.
Please read Part 1 of this article.
The key point to consider when changing an IP address is DNS. By its very nature, DNS information is distributed. That is part of how it works so well. If someone looks up the IP address for www.freebsddiary.org, they will store that answer for a while. That reduces network traffic, reduces the load on my DNS servers, and makes things faster at their computer (they don’t have to wait for each DNS query to complete; only the first one). It follows that the downside to this efficiency comes when the information must change. It takes time for this information to propagate. How much time depends upon your settings within your domain’s zone files. Look at the Expire and Refresh values to see just how long it can take. It appears that these values are not strictly honored by all DNS servers. Some like to cache longer than the domain settings. That’s their problem and there’s nothing you can do about that.
The box in question (m20.unixathome.org) has several functions, which include:
- mail server
- web server
- ssh server
Other IP addresses to change
I had other domains which pointed at m20. I wanted to update their DNS as well. Those changes simply involved modifying the IP address, as opposed to adding a second value to the existing host names. These were easy to do. After those changing had propagated, I was then ready to remove the old IP address from m20.
Swapping primary & alias on ifconfig – failure
On Monday, I decided it was time to go one step further in the process. I wanted to swap the primary and alias IP addresses. Although a NIC may have more than one IP address, by default, all outgoing connections will originate from the primary IP address and not the alias. This will have effects on the outside world as my box attempts to send out information. Namely, it will affect firewalls which [should] filter by IP address.
To effect these changes, I went ahead and modified
/etc/rc.conf to swap around the IP addresses which appeared in the
ifconfig_rl0_alias0 directives. Then I rebooted the box.
This was a disaster. The box never came back. And I had to drive into the POP to change things back. Of course, it
was just before close of business, and I also had somewhere to be. So rather than fully diagnose the problem, I
quickly changed the IP address information back to what it was before and rebooted. All was well. Things
were back online.
In hindsight, the messages on the console showed that routing was a problem. Here’s what I was seeing which led me to believe that:
I guessed that routing was a problem. Then I remembered that I had not changed the default route (i.e. the
Sep 30 16:09:05 m20 named: Ready to answer queries.
Sep 30 16:09:05 m20 ntpd: ntpd 4.1.0-a Thu Aug 1 00:21:33 EDT 2002 (1)
Sep 30 16:09:05 m20 ntpd: kernel time discipline status 2040
Sep 30 16:09:09 m20 ntpd: sendto(184.108.40.206): No route to host
Sep 30 16:09:17 m20 ntpd: sendto(220.127.116.11): No route to host
Sep 30 16:09:18 m20 ntpd: sendto(18.104.22.168): No route to host
Sep 30 16:09:18 m20 ntpd: sendto(22.214.171.124): No route to host
Sep 30 16:09:18 m20 ntpd: sendto(126.96.36.199): No route to host
Sep 30 16:09:22 m20 ntpd: sendto(188.8.131.52): No route to host
Sep 30 16:12:10 m20 lpd: Get_local_host: hostname 'm20.unixathome.org' bad
/etc/rc.conf). I wasn’t 100% positive that this was the only problem, but it looked like a starting point.
Swapping primary & alias on ifconfig – success
It wasn’t until Friday October 3 that I was ready to try again. This time I had a cunning plan. I had decided to change the IP addresses via the command line rather than risk a reboot. But this had to be done carefully. There was the risk that removing the IP address would terminate the ssh session. And then where would I be?
The plan involved two cron jobs: one to change the IP address, and another to reboot the box in case the change locked me out. A cunning plan indeed.
Here is the shell script:
Yes, those are the real IP addresses.
dan@m20:/usr/local/etc] $ less ifconfig.sh
ifconfig fxp0 inet 184.108.40.206 netmask 255.255.255.128
ifconfig fxp0 alias 220.127.116.11 netmask 255.255.255.255
route delete default
route add default 18.104.22.168
And here are the cron jobs:
36 * * * * root /usr/local/etc/ifconfig.sh
40 * * * * root /sbin/reboot
As you can see, at 36 past the hour, the IP addresses would change and so would the default gateway. That gave me four minutes to confirm everything was healthy before the system automagically rebooted.
After the shell script ran, I checked the NIC details:
Hmmm, well that certainly seems to have worked. I ran a few other tests (e.g. made sure I could create a new ssh session to the computer, and that I could run cvsup on the box and update the website).
$ ifconfig fxp0 fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet6 fe80::2a0:c9ff:fe49:baaa%fxp0 prefixlen 64 scopeid 0x1
inet 22.214.171.124 netmask 0xffffff80 broadcast 126.96.36.199
inet 188.8.131.52 netmask 0xffffffff broadcast 184.108.40.206
media: Ethernet autoselect (100baseTX <full-duplex>)
Don’t forget to disable those cron jobs!
Final /etc/rc.conf changes
The ultimate goal in swapping primary and alias IP addresses was to make it easier to remove the old IP address. That will
be done later using an
ifconfig fxp0 delete 220.127.116.11. But until that time comes, I really
/etc/rc.conf with the real information. But I’d rather do that tomorrow morning
rather than now (4:47 pm). That way, if something goes wrong, I can easily get access to the POP.
I’m not finished yet
There are still a few things left to do:
- reboot using the new
/etc/rc.conf(to ensure the changes work!)
- remove the old IP address from DNS
- let that change propogate
- remove the old IP address from fxp0 (and the alias from
- … I think that’s all
For those of us with home systems on DHCP, the address change is not simply magic and transparent. As mentioned in the first part of this series the firewall rules very likely have the local host address hard coded somewhere. Is possible to restart the firewall on DHCP address change. Am not totally warm and fuzzy with the hook I’ve added to my dhclient config.
But another thing I’ve found which doesn’t get mentioned is how some utilities, namely Network Time Daemon, ntpd, goes deaf when the local host IP address changes out from under it. Restarting this daemon is all it needs.
[x] You want to look type this:
I’ve always said use the tool which does the job. cron does the job. Why not use cron?
pretty simple… if you use cron you have to remember to remove the cron entry; at is better. Or perhaps even nohup…
How do you cancel an ‘at’ command? Oh, whoops… I guess you can use a kill command. Is there a way to do it via ‘at’?
At least if you *are* going to use cron, fill in ALL the fields in the thing! Is there any circumstance under which you would want this to run once an hour? No? Then specify an hour, and a day, and DOM, and month. Then you have a whole year to remove the entry, rather than that hour…
Just a quick hint – the general way of handling IP address propagation issues for changing the IP address of an A record is to lower the TTL value of that record PRIOR to making the change. It should be lowered enough in advance such that the older TTL will have expired anywhere it might be cached. The newer TTL should be something very low, like 5 minutes (or even less). This way, when you actually change the IP address of the record, you won’t have to wait 12 hours or whatever the old TTL was for all traffic to be using the new IP – everything will within 5 minutes or less using this method.
That’s good advice. I used that approach when moving FreshPorts from the old box to the new box.
Note that in the article, I am changing the IP address of the box. The services are not moving. I needed to have both IPs running in order to ensure that the new services where up and running and functional. I wanted to slowly move things from the old IP address to the new one in case I noticed any problems.
Regardless, reducing the TTL in my case would have been a good approach.