Nov 042010
 
DLT drive replacement

DLT drive replacement I am replacing a failed DLT 7000 drive with a known good drive. This page documents the steps, detailing the screws and their locations. I took these photos as I installed the drive, not as I removed it. Thus, some items may look out of place. I’ll present the photos in reverse order and how to simulate the removal. Remove the cover On the top of my Digital DLT MiniLibrary (Model DS-TL891-NE) has a single screw. Undo it and slide the cover towards the rear of the chassis. When replacing the cover, I needed to lift the front of the cover as I slide it foward. There are tabs which need to mesh on the rear of the case. The cover also has tabs underneath it which need to mesh with the front of the case. Two screws There are two screws to remove. This is the first one. The second one […]

Nov 032010
 
Replacing a DLT drive - selecting the right drive

Replacing a DLT drive – selecting the right drive The tape drive in my Digital DLT MiniLibrary (Model DS-TL891-NE) has died. Not totally dead. It just doesn’t respond to the library. It ends up with three flashing lights (Write Protected, Tape In Use, and Use Cleaning Tape) and the system can’t load a tape. Fortunately, I have more spare tape drives than I can use. Some time ago, I moved to disk based backups, with the plan to copy those backups to tape as well. When replacing one drive with another, you need to make sure they are both of similar types. In this case, they both need to be SCSI DIFF drives. The original drive: The replacement drive: The key points: Both are model TZ89N-AV Both are SCSI-DIFF Both have the same SCSI symbol (the graphic between SCSI and DIFF) It’s getting late, so now I’ll post this and then document the replacement procedure […]

Oct 152010
 

NRPE: Unable to read output – The followup Last week, I wrote about a problem with NRPE reporting NRPE: Unable to read output. Harold Paulson encountered the same issue yesterday. I was able to help him debug it today. We found a solution: a full path to sudo. We both had the same situation. Checking from the nagios server: $ /usr/local/libexec/nagios/check_nrpe2 -H kraken -c check_smartmon_ada8 NRPE: Unable to read output But when running locally on the Nagios client (NOTE: I amended the shell for nagios to /bin/sh for this test): # su -m nagios -c ‘sudo /usr/local/libexec/nagios/check_smartmon -d /dev/ada8′ OK: device is functional and stable (temperature: 35)|TEMP=35;55;60; If you don’t make that temporary shell adjustment, you’ll get this instead: # su -m nagios -c ‘sudo /usr/local/libexec/nagios/check_smartmon -d /dev/ada8′ UNKNOWN: no read permission given We couldn’t figure this out. Then, while looking at /usr/local/etc/nrpe.cfg and searching for sudo, I found: # *** THIS EXAMPLE MAY POSE […]

Oct 042010
 

NRPE: Unable to read output In a sudden flash of inspiration, I realized I could replace some cronjobs and emails with a Nagios plugin. This article assumes you know about Nagios and how to write a plugin. The main goal here is to show you a debugging approach I found. The problem The problem was the the following entry in /usr/local/etc/nrpe.cfg: command[check_freshports_queues]=/usr/local/libexec/nagios/check_freshports_queue When running it, I saw: $ /usr/local/libexec/nagios/check_nrpe2 -H nyi-vpn -c check_freshports_queues NRPE: Unable to read output A Google search shows this means the script it not producing any output. The script contains: #!/bin/sh cd /usr/websites/freshports.org/scripts/ ./queue-status.pl I started by adding an echo $? to the script. Now the output changes $ /usr/local/libexec/nagios/check_nrpe2 -H nyi-vpn -c check_freshports_queues 13 That’s the problem. The script isn’t running at all. No error output though. Let me try another trick. On the remote server, become root, and then run the command from nrpe.cfg via su. This allows you […]

Sep 222010
 

ZFS: promote – for getting rid of snapshots, but not the data When I built the Kraken ZFS file server, I used a snapshot to copy the data over. Those snapshots are still hanging around. I’d like to get rid of them. I can do that with zfs promote. This article documents how I did that. The background I have a few snaphots here? $ zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT storage/Retored@2010.07.27 706G – 3.11T – storage/Retored@2010.07.28 264K – 2.42T – Those snapshots were used with zfs send and zfs receive to copy the files from the original ZFS system to the new one. Now that array is solid and stable, the snapshots are still around but now longer needed. My goal is to get rid of them. On a side note, I am annoyed by this inconsistency in free space: $ zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT storage […]

Sep 202010
 

ZFS: upgrade Yesterday I upgraded my file server to the latest version of FreeBSD (8.1-STABLE #0: Sat Sep 18 23:43:48 EDT 2010) by building world and installing. My goal was to get a recently MFC’d feature related to scrubbing of pools. I tried to find the original commit message but failed. But Revision 211800 seems related. After upgrading, I then added daily_scrub_zfs_enable=”YES” to /etc/periodic.conf. The next morning, I found this in my ‘daily run output’ email: Checking status of zfs pools: pool: storage state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using ‘zpool upgrade’. Once this is done, the pool will no longer be accessible on older software versions. scrub: scrub in progress for 6h51m, 39.36% done, 10h33m to go config: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 0 raidz2 ONLINE 0 0 0 […]

Sep 032010
 

3Ware Nagios plugin I use Nagios to monitor my servers and work stations. If something goes wrong, I usually get told by Nagios before I notice the problem myself. A week or so back, I noticed a rather odd RAID problem. Eventually, the problem was solved by upgrading the firmware on the controller. In the meantime, I had located and installed a Nagios 3ware plugin. I like it and I’m using it on more than one server. However, now that I turned on AUTO-VERIFY, I’ve found a spot where I can improve the plugin. Verifying…! Earlier today, I turned on AUTO-VERIFY for this controller. Tonight, Nagios is reporting: Status: UNKNOWN Status Information: UNKNOWN: /c0/u0 RAID-10 VERIFYING – 56% 64K 195.548 ON ON – /c0/u1 SPARE VERIFYING – 0% – 69.2404 – ON – /c0/u2 SPARE VERIFYING – 0% – 69.2404 – ON – If I look at the status output, I see: $ sudo /usr/local/sbin/tw_cli […]

Aug 302010
 

Where is my SCSI card? At the start of the month, Jerry was over helping me to add more HDD to my Bacula Storage Daemon. As part of that process, I moved my tape library from my Bacula server over to the Storage Daemon. This required the transfer of a SCSI card from one machine to another. Jerry helped with that too. At that time, I didn’t both to check the card or the library. Yesterday, I noticed the SCSI card (an Adaptec AHA-2944UW) was not appearing in the dmsg(8) output. However, it turned out to be easy to fix. Watching the boot process While watching the boot process, I saw this message: Adaptec AHA-2944UW Differential SCSI BIOS v 2.20.0 (c) 1999 Adaptec, Inc. All Rights Reserved. Can’t Locate Host Adapter SCSI BIOS Not Installed! It was suggested that I try moving the card to another slot. That would have to wait for another day. […]

Aug 042010
 

ZFS: Resizing your zpool I’m about to rebuild my ZFS array (which I documented in my other diary). The array has been running for a while, but I recently learned some new facts about ZFS which spurred me on to rebuilding my array with future-proofing in mind. This is my plan for tonight. As I type this, Jerry is over tonight, doing the heavy lifting for me. I am nursing a broken left elbow. The two new HDD have been installed and the system has been powered back up. Tonight we will do the following: identify the newly installed HDD put a file system on those HDD copy the existing ZFS array over to that new FS (call this temp) destory the existing ZFS array parition each individual drive using gpart add the drives back into the array copy the data back partition the two new FS and put them into the new array My […]

Aug 012010
 

ZFS: do not give it all your HDD I’m about to rebuild my ZFS array (which I documented in my other diary). The array has been running for a while, but I recently learned some new facts about ZFS which spurred me on to rebuilding my array with future-proofing in mind. This is my plan for tonight. As I type this, Jerry is over tonight, doing the heavy lifting for me. I am nursing a broken left elbow. The two new HDD have been installed and the system has been powered back up. Tonight we will do the following: identify the newly installed HDD put a file system on those HDD copy the existing ZFS array over to that new FS destroy the existing ZFS array partition each individual drive using gpart add the drives back into the array copy the data back partition the two new FS and put them into the new array […]