Adding gmirror to an existing installationThis article describes how I created a RAID-1 cluster on my FreeBSD 7.0-STABLE box using gmirror. Back in March, I bought an AMD Phenom(tm) and starting putting it together. You can read more about the box and the dual monitor setup here:
- jail server
- Regression testing for Bacula
- Downloading database backups and test loading them (FreshPorts, BSDCan, etc)
FreeBSD is already installedOn this system, FreeBSD is already installed. Usually, one sets up gmirror when installing the base system. In this case, I will boot from a FreeBSD 7.0-RELEASE LIVE CD and use that to setup the RAID-1 cluster.
Identify the HDDWhen setting up gmirror, you will tell it to use one HDD as the master. The other will be a slave. These are not the correct technological terms, but they will do for my purposes. The critical part of this process is making sure that you select the correct HDD for the master. If not, you will lose your installed system. My cunning plan to correctly identify the HDD was simple. Boot from the HDD and create a distinctly unique filename in the root directory. Then boot from the live CD, mount the HDD, and find the file. Simple. What could go wrong?
Starting the live CDThe ISO image I downloaded was 7.0-RELEASE-amd64-livefs.iso. The livefs part is the key. You boot this like any other CD. From the main menu, select Fixit, then CDROM/DVD. You should then see a Fixit# prompt.
Find the right HDDUsing dmesg I found I had two HDD mounted, ad4 and ad6. Let’s try mounting them and see what I can find:
Fixit# mount /dev/ad4s1 /mnt Fixit# ls /mnt . home .. lib .cshrc libexec .profile media .snap mnt COPYRIGHT mnt2 THIS.IS.6.3.BLOW.It.away proc bin rescue boot root cdrom sbin compat sys dev tmp dist usr entropy var etcNOTE: that’s not the actual directory listing output. It is made up. The key is that odd looking file name. It identifies the HDD that I do not wish to keep. All data on this HDD can be lost. Therefore, I know that ad4 is the slave HDD. Now let’s try the other HDD and see what it contains:
Fixit# umount /mnt Fixit# mount /dev/ad6s1 /mnt Fixit# ls /mnt .cshrc boot/ entropy/ media/ sbin/ .profile cdrom/ etc/ mnt/ sys@ .snap/ compat@ home@ proc/ tmp/ COPYRIGHT dev/ lib/ rescue/ usr/ bin/ dist/ libexec/ root/ var/ Fixit#Again, this is not a copy/paste. It may not represent exactly what I saw. The key point: there is no special file. I know that ad6 is the master HDD to use in the gmirror solution.
Setting up the mirror/duplexI have before me, a copy of The Best of FreeBSD Basics by Dru Lavigne. I used the section Using Software RAID-1 With FreeBSD. This is not the first time I’ve used this documentation. I previously upgraded my gateway/firewall box to use gmirror. The first step is to create a mirror/duplex. It is critical that you specify the drive that you wish to mirror, i.e. the master drive. In my case, as found above, that drive is /dev/ad6.
Fixit# gmirror label -v -b round-robin gm0 /dev/ad6 Metadata value stored on /dev/ad6 DoneThis created gm0 and uses /dev/ad6 as the master for this mirror. At this stage, the data has not yet been mirrored. All we have done is specify that we want to mirror.
Mount the main HDD for editingI will now mount /dev/ad6s1 so I can alter a few files.
Fixit# mount /dev/ad6s1 /mnt
Start gmirror at bootThe following line was added to /mnt/boot/loader.conf to ensure gmirror started at boot time:
Alter fstabInstead of mounting /dev/ad6* at the next boot, we want to mount /dev/mirror/gm0. This is how FreeBSD will access your HDD, not directly, but through the mirror. This is similarly to how hardware RAID works; the RAID cards presents a device to the OS, and the OS accesses that device, e.g. twed0. I will now alter /mnt/etc/fstab. I will change all ad6 references to mirror/gm0
Fixit# vi /mnt/etc/fstabThis vi command does a global replacement:
:%s/ad6/gmirror\/gm0/gHere is what fstab looks like now:
# Device Mountpoint FStype Options Dump Pass# /dev/mirror/gm0s1b none swap sw 0 0 /dev/mirror/gm0s1a / ufs rw 1 1 /dev/mirror/gm0s1e /tmp ufs rw 2 2 /dev/mirror/gm0s1f /usr ufs rw 2 2 /dev/mirror/gm0s1d /var ufs rw 2 2 /dev/acd0 /cdrom cd9660 ro,noauto 0 0
BootingI then rebooted the system. However, it booted from ad4 instead of ad6. That’s a problem. It needs to boot from ad6. I was able to fix this in the BIOS. But not before playing around and trying to find a solution from the boot loader prompt. I found this would work:
After bootingAfter getting the system booted from the correct HDD, things went much smoother. So far, we have set up the mirror and specified the source for the mirror. Now we want to add the other half of the mirror into the system. That other drive is /dev/ad4. I used this command:
gmirror insert gm0 /dev/ad4[My thanks to manolis for pointing this out. After adding the new disk to the mirror, here is the list of the gmirror devices:
# gmirror list Geom name: gm0 State: DEGRADED Components: 2 Balance: round-robin Slice: 4096 Flags: NONE GenID: 0 SyncID: 1 ID: 4226236689 Providers: 1. Name: mirror/gm0 Mediasize: 500107861504 (466G) Sectorsize: 512 Mode: r6w5e6 Consumers: 1. Name: ad6 Mediasize: 500107862016 (466G) Sectorsize: 512 Mode: r1w1e1 State: ACTIVE Priority: 0 Flags: DIRTY GenID: 0 SyncID: 1 ID: 533080279 2. Name: ad4 Mediasize: 500107862016 (466G) Sectorsize: 512 Mode: r1w1e1 State: SYNCHRONIZING Priority: 0 Flags: DIRTY, SYNCHRONIZING GenID: 0 SyncID: 1 Synchronized: 0% ID: 438208982 #Yes, ad6 is dirty. That’s to be expected. At any given point, there is probably some data to be written to disk. That’s all that means. Wait a while. Try again. Perhaps it won’t be dirty. You will see that ad4 is synchronizing. That took a few hours. These are 500GB HDD. The status command gives a more succinct output
# gmirror status Name Status Components mirror/gm0 DEGRADED ad6 ad4 (6%)
After the syncOnce everything is synced up, and the initial mirroring process has completed, this is what I saw:
# gmirror status Name Status Components mirror/gm0 COMPLETE ad4 ad6
The step where the slave disk is inserted into the mirror, something like:
gmirror insert gm0 /dev/ad4
you may wish to show this too…
The Man Behind The Curtain