Converting a system to RAID
This may well be the last article I write on this server. My existing
development machine is still on 4.11. Today I will start the migration process
to another server, which is already running 6.2-STABLE. In the process, I will
add a RAID card to the server, and transfer the existing OS from a SATA drive to
the RAID-1 cluster (based on two 160GB IDE drives).
Note that although I am using a RAID array here, it appears to the OS as a single HDD,
so this process could equally apply to moving to a new bigger HDD. For your information,
I will be using a 3Ware 7006-2
from 3Ware.
The main reason for putting RAID into this machine is that it will become my main development machine
and take over the duties of two machines that now run FreeBSD 4.11 (which has recently hit End-of-Life
and no more security updates will occur for 4.x).
We now join the migration, already in progress.
Is the RAID card there?
As of this point, I’ve installed the two new IDE drives and hooked them up to the 3Ware card.
fter boot, I see this in the output from dmesg:
twe0: [GIANT-LOCKED] twe0: 2 ports, Firmware FE7X 1.05.00.063, BIOS BE7X 1.08.00.048
That is my RAID card, just waiting to be set up. Good. Now I will reboot and
go through the RAID BIOS and set up the cluster.
By the way, I’ve also installed sysutils/tw_cli and sysutils/3dm, two 3Ware utilities.
I have mentioned them in a previous article on the 3Ware 9550SX card.
Preparing the system for the transfers
We will be using FreeSBIE to do the data transfer between the old and new HDD. Why? It is easier to copy an HDD
when it is unused and with no chance of the data being modified. In short, the system is offline and copying
the HDD is guaranteed to be correct.
From there, I used the instructions for Swapping boot drives around, written
way back in August 1999. All I needed was the details for tar’ing up one file system and untar’ing it on the other.
For this clean and easy technique, I will boot from something other than the SATA drive
and the two IDE RAID drives. I could have used an HDD which already had FreeBSD on it.
I choose instead to use a FreeSBIE live CD. FreeSBIE 2.0 gives me FreeBSD
6.2 and all the tools I need to massage the data according to my will.
First, I boot up the system from the CD. Then I su to root, and move to my home directory.
Then I create two directories:
mkdir new old
These two directories will form the starting point for my transfers. I will mount the existing SATA drive
under the old directory and the new RAID-1 system under the new directory.
I need mount points for the various slices. So I did this:
cd old mkdir root var tmp usr cd .. cd new mkdir root var tmp usr
As you will see in the problems section later on, I forgot to do this, but you won’t:
chmod 1777 ~/old/tmp
I started sysinstall and then used it to configure the new HDD (twed0). I click on the following links
once in sysinstall:
- Configure | Fdisk | twed0 | A = Use Entire Disk | W = Write Changes | Yes |
Standard Install a standard MBR (no boot manager)
That will run fdisk on the new HDD. When completed , press Q – Quit and then Cancel
to get back to the FreeBSD Configuration Menu
I will now label the new HDD. Again, from the FreeBSD Configuration Menu, I click on Label.
Ensure the right disk is selected, twed0 in my case.
Now I will create several slices and assign them some mount points. sysinstall will actually
mount the slices at the mount points specified, and which we created earlier in this process.
The short hand here is:
C = Create | 1 GB | FS | /mnt/root C = Create | 2 GB | swap C = Create | 2 GB | FS | /mnt/tmp C = Create | 2 GB | FS | /mnt/var C = Create | All (142GB) | FS | /mnt/usr W = Write Yes
Here is what the screen looked like after issuing the Write command.
FreeBSD Disklabel Editor Disk: twed0 Partition name: twed0s1 Free: 0 blocks (0MB) Part Mount Size Newfs Part Mount Size Newfs ---- ----- ---- ----- ---- ----- ---- ----- twed0s1a /mnt/root 1024MB UFS2+S Y twed0s1b swap 2048MB SWAP twed0s1e /mnt/tmp 2048MB UFS2+S Y twed0s1f /mnt/var 2048MB UFS2+S Y twed0s1g /tmp/usr 142GB UFS2+S Y
And in case it helps you, here are the the mounted filesystems:
[dan@ngaio:/mnt] $ mount /dev/ad6s1a on / (ufs, local) devfs on /dev (devfs, local) /dev/ad6s1e on /tmp (ufs, local, soft-updates) /dev/ad6s1f on /usr (ufs, local, soft-updates) /dev/ad6s1d on /var (ufs, local, soft-updates) fdescfs on /dev/fd (fdescfs) polo:/usr/ports/distfiles on /usr/ports/distfiles (nfs) /dev/twed0s1a on /mnt/root (ufs, local, soft-updates) /dev/twed0s1e on /mnt/tmp (ufs, local, soft-updates) /dev/twed0s1f on /mnt/var (ufs, local, soft-updates) /dev/twed0s1g on /tmp/usr (ufs, local, soft-updates) [dan@ngaio:/mnt] $
I will not be copying over devfs or fdescfs. NFS mounts should also
not be copied over. In short, look only for the mount points associated
with the HDD in question. In this case, ad6.
Of note were some interesting directories I had never seen before:
[dan@ngaio:/mnt] $ find . . ./tmp ./tmp/.snap ./var ./var/.snap ./root ./root/.snap ./usr [dan@ngaio:/mnt] $
What are those files? A Google found me a answer.
In short, see man dump(8)
and read the -L option.
Here is how I mounted the various slices of the SATA drive:
mount /dev/ad6s1a ~/old/root mount /dev/ad6s1e ~/old/tmp mount /dev/ad6s1f ~/old/var mount /dev/ad6s1d ~/old/usr
Similarly, for the new RAID-1 cluster:
mount /dev/twed0s1d ~/new/root mount /dev/twed0s1e ~/new/tmp mount /dev/twed0s1f ~/new/var mount /dev/twed0s1g ~/new/usr
Everything is ready. Let’s transfer the data. I will do this one file system at a time. Let’s
start with the root partition:
tar --one-file-system -c -f - -C ~/old/root/ . | tar xpvf - -C ~/new/root/
The tar on the left will do a cd to ~/old/root/ and start tar’ing everything up, staying
within the one file system. The -c is to create. The -f designates the output file, this case
STDOUT.
The tar on the right will take extract, retain permissions, and display each file name as it is extracted.
The -f – indicates STDIN as the source of the tarball. tar will do a cd to ~/new/root before it starts
this process.
This process needs to be repeated for each mount point:
tar --one-file-system -c -f - -C ~/old/var/ . | tar xpvf - -C ~/new/var/ tar --one-file-system -c -f - -C ~/old/tmp/ . | tar xpvf - -C ~/new/tmp/ tar --one-file-system -c -f - -C ~/old/usr/ . | tar xpvf - -C ~/new/usr/
That sounds just too easy. Well, there is one gotcha. If anything refers to the old HDD
by device, then it will break. You must change the new /etc/fstab before you reboot from the
new RAID array. In my case, the file I need to edit is ~/new/root/etc/fstab and the changes required
reflect what happened when I was using the Disk Label Editor. That is, I made sure that /dev/twed0s1a
was mounted at /, twed0s1b was swap, twed0s1e was mounted at /tmp, etc.
Then I shutdown the system with a shutdown -p now and disconnected the SATA drive.
During the power up process, I removed the FreeSBIE CD from the CD drive and let the system
boot from the RAID card. Apart from the minor permission problem mentioned below, it went
flawlessly.
The /tmp permission problem
The system would not boot from the RAID cluster the first time I tried. The problem was I had root on /dev/twed0s1d instead of /dev/twed0s1a.
I went into FreeSBIE and rebooted. I went into the Disk Label Editor, deleted the slice, and created a new one,
this time with s1a, not s1d. I had to redo the tar | tar process, but that didn’t take too long. I also had
to edit ~/new/root/etc/fstab to point at twed0 instead of ad6.
The next reboot worked, but there was a problem with /tmp permissions. A chmod 1777 fixed that. The next reboot
showed no problems.