Posted on May 13, 2009 under nexenta, opensolaris, zfs |
Hybrid storage pools are the new coolest thing in ZFS land and are being pushed commercially with the recent release of Sun’s 7000 series open storage. For various reasons, Sun has only added a supported SSD option to the X4540 Thor systems. This unfortunately doesn’t help those of us wanting to extend our X4500 Thumper investments and get in on the SSD performance action. I’m fairly certain you could buy Sun’s Thor solution and use it in your Thumper, but I for one wasn’t impressed with the price.
Fortunately there is a simple, relatively inexpensive way to use SSD drives with your Thumper. Here’s what you need:
You’ll also want a spare X4500 drive bracket unless you’re willing to use a bracket from the drive(s) you have to pull out. Insert your Intel X25-E SSD into the Icy Dock, attach the Icy Dock to the X4500 bracket like any 3.5″ drive, and insert the result into your Thumper. Voila. The Icy Dock bay is well constructed despite being plastic (I would prefer it was made out of aluminum). Also, $20 for a hunk of plastic and a converter isn’t ideal, but those of you that would suggest duct tape have no sense of aesthetics.
In my case, I swapped out two 1TB drives for two 32GB Intel X25-E SSDs that are used as mirrored ZFS intent log. They show up and work just fine:
platform = Sun Fire X4500
Device Serial Vendor Model Rev Temperature
—— —— —— —– —- ———–
c0t0d0p0 F002PAJSZPLF ATA HITACHI HUA7210S A90A None
c0t1d0p0 F002PAJSX74F ATA HITACHI HUA7210S A90A None
c0t2d0p0 F002PAJT81VF ATA HITACHI HUA7210S A90A None
c0t3d0p0 F002PAJSYHMF ATA HITACHI HUA7210S A90A None
c0t4d0p0 F002PAJSZUNF ATA HITACHI HUA7210S A90A None
c0t5d0p0 F002PAJSATHF ATA HITACHI HUA7210S A90A None
c0t6d0p0 F002PAJSXA6F ATA HITACHI HUA7210S A90A None
c0t7d0p0 F002PAJSX7XF ATA HITACHI HUA7210S A90A None
c1t0d0p0 F002PAJSVSHF ATA HITACHI HUA7210S A90A None
c1t1d0p0 F002PAJSTBEF ATA HITACHI HUA7210S A90A None
c1t2d0p0 F002PAJSVAJF ATA HITACHI HUA7210S A90A None
c1t3d0p0 F002PAJT5XDF ATA HITACHI HUA7210S A90A None
c1t4d0p0 F002PAJSZNXF ATA HITACHI HUA7210S A90A None
c1t5d0p0 F002PAJPZ8DF ATA HITACHI HUA7210S A90A None
c1t6d0p0 F002PAJSVBEF ATA HITACHI HUA7210S A90A None
c1t7d0p0 F002PAJSV9XF ATA HITACHI HUA7210S A90A None
c3t0d0p0 u SanDisk U3 Cruzer Micro 8.02 None
c4t0d0p0 u SanDisk U3 Cruzer Micro 8.02 None
c5t0d0p0 F002PAJSK27F ATA HITACHI HUA7210S A90A None
c5t1d0p0 F002PAJSVSRF ATA HITACHI HUA7210S A90A None
c5t2d0p0 F002PAJKZNAF ATA HITACHI HUA7210S A90A None
c5t3d0p0 F002PAJSXKPF ATA HITACHI HUA7210S A90A None
c5t4d0p0 F002PAJSYB9F ATA HITACHI HUA7210S A90A None
c5t5d0p0 F002PAJSVYDF ATA HITACHI HUA7210S A90A None
c5t6d0p0 F002PAJSAUUF ATA HITACHI HUA7210S A90A None
c5t7d0p0 F002PAJSSWYF ATA HITACHI HUA7210S A90A None
c6t0d0p0 140085032HGN ATA SSDSA2SH032G1GN 8790 None
c6t1d0p0 F002PAJSZGMF ATA HITACHI HUA7210S A90A None
c6t2d0p0 F002PAJTD4WF ATA HITACHI HUA7210S A90A None
c6t3d0p0 F002PAJSV2SF ATA HITACHI HUA7210S A90A None
c6t4d0p0 1400CB032HGN ATA SSDSA2SH032G1GN 8790 None
c6t5d0p0 F002PAJSJMTF ATA HITACHI HUA7210S A90A None
c6t6d0p0 F002PAJRZXUF ATA HITACHI HUA7210S A90A None
c6t7d0p0 F002PAJSSVLF ATA HITACHI HUA7210S A90A None
c7t0d0p0 F002PAJSX9YF ATA HITACHI HUA7210S A90A None
c7t1d0p0 F002PAJSZNSF ATA HITACHI HUA7210S A90A None
c7t2d0p0 F002PAJSLE3F ATA HITACHI HUA7210S A90A None
c7t3d0p0 F002PAJSZ3HF ATA HITACHI HUA7210S A90A None
c7t4d0p0 F002PAJSVH3F ATA HITACHI HUA7210S A90A None
c7t5d0p0 F002PAJSVGMF ATA HITACHI HUA7210S A90A None
c7t6d0p0 F002PAJPWV8F ATA HITACHI HUA7210S A90A None
c7t7d0p0 F002PAJSY12F ATA HITACHI HUA7210S A90A None
c8t0d0p0 F002PAJS0XEF ATA HITACHI HUA7210S A90A None
c8t1d0p0 F002PAJSVZ0F ATA HITACHI HUA7210S A90A None
c8t2d0p0 F002PAJSZ46F ATA HITACHI HUA7210S A90A None
c8t3d0p0 F002PAJSN68F ATA HITACHI HUA7210S A90A None
c8t4d0p0 F002PAJSWM6F ATA HITACHI HUA7210S A90A None
c8t5d0p0 F002PAJPZ8BF ATA HITACHI HUA7210S A90A None
c8t6d0p0 F002PAJSVGTF ATA HITACHI HUA7210S A90A None
c8t7d0p0 F002PAJS1AJF ATA HITACHI HUA7210S A90A None
———————SunFireX4500——Rear—————————-
36: 37: 38: 39: 40: 41: 42: 43: 44: 45: 46: 47:
c6t3 c6t7 c5t3 c5t7 c8t3 c8t7 c7t3 c7t7 c1t3 c1t7 c0t3 c0t7
^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++
24: 25: 26: 27: 28: 29: 30: 31: 32: 33: 34: 35:
c6t2 c6t6 c5t2 c5t6 c8t2 c8t6 c7t2 c7t6 c1t2 c1t6 c0t2 c0t6
^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++
12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23:
c6t1 c6t5 c5t1 c5t5 c8t1 c8t5 c7t1 c7t5 c1t1 c1t5 c0t1 c0t5
^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++
0: 1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11:
c6t0 c6t4 c5t0 c5t4 c8t0 c8t4 c7t0 c7t4 c1t0 c1t4 c0t0 c0t4
^b+ ^b+ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++ ^++
——-*———–*-SunFireX4500–*—Front—–*———–*———-
Summary:
Vendor Model Count
—— —– —–
ATA HITACHI HUA7210S 46
SanDisk U3 Cruzer Micro 2
ATA SSDSA2SH032G1GN 2
Total Storage Devices = 50
To help make up for the lost space, I also use two SanDisk Cruzer Micro 16GB flash drives as a mirrored ZFS boot/root. This works really well under Nexenta NCP and NexentaStor.
Posted on Feb 24, 2009 under nexenta, opensolaris |
The current Nexenta NCP 2.0 hardy-unstable repository has some broken bits that prevent a clean apt-clone dist-upgrade from working. A common question on the IRC channel is how to work around this, and I’ve answered it enough times that I thought it best to write up some basic instructions.
Here’s an example of what happens:
# apt-clone -v dist-upgrade
This operation will upgrade your system using ZFS capabilities. Proceed ? (y/n) y
Updating APT sources …
apt-get -V update
Hit http://apt.nexenta.org hardy-unstable Release.gpg
Hit http://apt.nexenta.org hardy-unstable Release
Hit http://apt.nexenta.org hardy-unstable/main Packages
Hit http://apt.nexenta.org hardy-unstable/contrib Packages
Hit http://apt.nexenta.org hardy-unstable/non-free Packages
Hit http://apt.nexenta.org hardy-unstable/main Sources
Hit http://apt.nexenta.org hardy-unstable/contrib Sources
Hit http://apt.nexenta.org hardy-unstable/non-free Sources
Reading package lists… Done
Initiating Nexenta upgrade procedure. Please wait…
Success. This upgrade will download approximately 38.90MB
This upgrade will require REBOOT. Proceed? (y/n) y
Upgrade is in progress. Please DO NOT interrupt…
Creating Upgrade Checkpoint…
syspool/rootfs-nmu-002: entry not found in menu.lst
Upgrade Checkpoint has been created: rootfs-nmu-002
Use ‘zfs list -r syspool’ command to list all available
upgrade/rollback checkpoints
apt-get -V -y install dpkg apt
Reading package lists… Done
Building dependency tree
Reading state information… Done
dpkg is already the newest version.
apt is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 16 not upgraded.
rmdir /tmp/upgrade-attempt.21883 2>/dev/null
mkdir /tmp/upgrade-attempt.21883
mount -F zfs syspool/rootfs-nmu-002 /tmp/upgrade-attempt.21883 2>/dev/null
chroot /tmp/upgrade-attempt.21883 mount /proc 2>/dev/null
mount -F lofs -O /etc/mnttab /tmp/upgrade-attempt.21883/etc/mnttab 2>/dev/null
mount -F lofs -O /devices /tmp/upgrade-attempt.21883/devices 2>/dev/null
APT_CLONE_ENV=1 chroot /tmp/upgrade-attempt.21883 apt-get -V -o Dpkg::Options::=”–force-confdef” -o Dpkg::Options::=”–force-confold” dist-upgrade
Reading package lists… Done
Building dependency tree
Reading state information… Done
Calculating upgrade… Done
The following packages will be upgraded:
man-db (2.5.2-2nexenta1 => 2.5.2-2nexenta2)
sunwcakr (5.11.104-2 => 5.11.104-3)
sunwckr (5.11.104-4 => 5.11.104-5)
sunwcsl (5.11.104-5 => 5.11.104-6)
sunwemlxs (5.11.104-1 => 5.11.104-2)
sunwiscsidmr (5.11.104-1 => 5.11.104-2)
sunwiscsidmu (5.11.104-1 => 5.11.104-2)
sunwiscsir (5.11.104-5 => 5.11.104-6)
sunwiscsitr (5.11.104-1 => 5.11.104-2)
sunwiscsitu (5.11.104-1 => 5.11.104-2)
sunwkrbu (5.11.104-1 => 5.11.104-2)
sunwstmf (5.11.104-1 => 5.11.104-2)
sunwstmfu (5.11.104-1 => 5.11.104-2)
sunwzfskr (5.11.104-5 => 5.11.104-7)
sunwzfsr (5.11.104-4 => 5.11.104-6)
sunwzfsu (5.11.104-4 => 5.11.104-6)
16 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 20.8MB/38.9MB of archives.
After this operation, 3048kB disk space will be freed.
Do you want to continue [Y/n]? y
Get:1 http://apt.nexenta.org hardy-unstable/main sunwcakr 5.11.104-3 [2462kB]
Get:2 http://apt.nexenta.org hardy-unstable/main sunwckr 5.11.104-5 [14.5MB]
Get:3 http://apt.nexenta.org hardy-unstable/main sunwzfsr 5.11.104-6 [453kB]
Get:4 http://apt.nexenta.org hardy-unstable/main man-db 2.5.2-2nexenta2 [1050kB]
Get:5 http://apt.nexenta.org hardy-unstable/main sunwiscsir 5.11.104-6 [236kB]
Get:6 http://apt.nexenta.org hardy-unstable/main sunwzfskr 5.11.104-7 [851kB]
Get:7 http://apt.nexenta.org hardy-unstable/main sunwzfsu 5.11.104-6 [1253kB]
Fetched 20.8MB in 49s (420kB/s)
nl.1: sh: fatal: relocation error: file /lib/libncurses.so.5: symbol main: referenced symbol not found
E: Sub-process /usr/bin/apt-clone -c exited unexpectedly
E: Failure running script /usr/bin/apt-clone -c
chroot /tmp/upgrade-attempt.21883 umount /proc 2>/dev/null
umount /tmp/upgrade-attempt.21883/etc/mnttab 2>/dev/null
umount /tmp/upgrade-attempt.21883/devices 2>/dev/null
rm -rf /tmp/nl*
umount /tmp/upgrade-attempt.21883 2>/dev/null
rmdir /tmp/upgrade-attempt.21883
Upgrade failed. Would you like to rollback changes now? (y/n)
In order to fix this, you would answer no to the rollback and then execute something like this:
# mount -F zfs syspool/rootfs-nmu-002 /mnt
# chroot /mnt mount /proc
# mount -F lofs -O /etc/mnttab /mnt/etc/mnttab
# mount -F lofs -O /devices /mnt/devices
# chroot /mnt
Now from within the chroot, execute:
# rm -r /tmp/nl*
# APT_CLONE_ENV=1 apt-get -V -o Dpkg::Options::=”–force-confdef” -o Dpkg::Options::=”–force-confold” dist-upgrade
# bootadm update-archive
# exit
This should get all the packages within the clone upgraded properly, update the boot archive, and back you out of the clone. Now you need to install a new grub, unmount the clone, and reboot:
# installgrub /mnt/boot/grub/stage1 /mnt/boot/grub/stage2 /dev/rdsk/
# installgrub /mnt/boot/grub/stage1 /mnt/boot/grub/stage2 /dev/rdsk/ (if you have one)
# chroot /mnt umount /proc
# umount /mnt/devices
# umount /mnt/etc/mnttab
# umount /mnt
Obviously you’ll need to replace with the correct device(s), such as c0d0s0 and c1d0s0. If you don’t know which devices to use, run zpool status syspool and it should tell you. Note above that we did not activate the new clone as the default bootfs. It is best now to reboot, select the new entry from the grub menu, and verify that the clone is working properly. Once you’re booted into the clone, you can run apt-clone -a rootfs-nmu-002 to make it the default.
Hopefully this helps some of you out there having problems. Feel free to drop by ##nexenta on irc.freenode.net if you require more assistance.
Posted on Dec 16, 2008 under nexenta, opensolaris, zfs |
Every self-respecting geek has a home media server to store their CD and DVD rips, downloaded TV shows, photos, etc. and I’m no exception. I’ve always found it interesting to see what others put together to fill this void, so I’m returning the favor by showing off my current setup:
The full components list includes:
platinum.alloy.net serves up audio/video/photos/etc. to two hacked AppleTV boxes, my wife’s iMac, and my Mac Pro via Samba. It also serves up home directories for the Sun Ray thin-client in the kitchen and acts as a backup server for our Macbook Pro laptops.
It has been through a number of incarnations over the years. I picked up the Chenbro case off eBay a year or so ago for around $350 which was a steal, considering they run upwards of $800 new. The motherboard, CPU, memory, and 1TB drives are new as of December 2008 (early Christmas present, yay!). Everything else I’ve had for a couple of years and it’s been the usual organic upgrade process. The Supermicro MBD-X8SAX-O motherboard is so new that I must have been the first one to buy it off Newegg. The price for the new motherboard/cpu/memory dropped about $100 less than a week after I ordered, and I hadn’t even received the parts yet. Typical! I tend not to be so bleeding edge and only upgrade every three years or so, but the timing was right and Nehalem was calling me in my sleep.
I was previously using a Tyan S2865G2NR board with a 2.2GHz AMD Athlon X2 and 4GB of Crucial memory. There was nothing wrong with this other than it didn’t have enough PCI-X or PCI-E slots, and socket 939 is getting a bit long in the tooth. The upshot is that I can get ~$200 out of the old parts on eBay and recoup some of the bleeding edge cost.
Upgrading to the new motherboard was mostly straightforward, although I had to do a manual dist-upgrade to the latest NexentaCP unstable since the OpenSolaris b85 kernel had problems booting. I burned NexentaCore 2.0 Alpha1 and used the recovery grub option to login, mount my syspool, and fix things. I’m in the process of migrating all the data off the six Seagate 250GB drives which are attached to the AOC-SAT-MV8. This card has had a lot of problems with ZFS and has been discontinued by Supermicro in favor of the AOC-SAT2-MV8. Whenever I expand beyond six disks, I’m going to skip straight to the AOC-USAS-L8i using mini-SAS to SATA cables.
So far the new board/cpu/memory appear stable, and it’s been serving up WALL-E repeatedly to our three year old for a few days now. It’s nice to see Hyperthreading giving us what appears as 8 core bling:
# psrinfo -v
Status of virtual processor 0 as of: 12/15/2008 22:56:39
on-line since 12/14/2008 18:22:47.
The i386 processor operates at 2660 MHz,
and has an i387 compatible floating point processor.
Status of virtual processor 1 as of: 12/15/2008 22:56:39
on-line since 12/14/2008 18:22:51.
The i386 processor operates at 2660 MHz,
and has an i387 compatible floating point processor.
Status of virtual processor 2 as of: 12/15/2008 22:56:39
on-line since 12/14/2008 18:22:51.
The i386 processor operates at 2660 MHz,
and has an i387 compatible floating point processor.
Status of virtual processor 3 as of: 12/15/2008 22:56:39
on-line since 12/14/2008 18:22:51.
The i386 processor operates at 2660 MHz,
and has an i387 compatible floating point processor.
Status of virtual processor 4 as of: 12/15/2008 22:56:39
on-line since 12/14/2008 18:22:51.
The i386 processor operates at 2660 MHz,
and has an i387 compatible floating point processor.
Status of virtual processor 5 as of: 12/15/2008 22:56:39
on-line since 12/14/2008 18:22:51.
The i386 processor operates at 2660 MHz,
and has an i387 compatible floating point processor.
Status of virtual processor 6 as of: 12/15/2008 22:56:39
on-line since 12/14/2008 18:22:51.
The i386 processor operates at 2660 MHz,
and has an i387 compatible floating point processor.
Status of virtual processor 7 as of: 12/15/2008 22:56:39
on-line since 12/14/2008 18:22:51.
The i386 processor operates at 2660 MHz,
and has an i387 compatible floating point processor.
I’ll work on benchmarking this home-grown thumper after the New Year. The next upgrade is to install four Intel X25-M SSDs for the ZIL and L2ARC caches. Given the cost I think I’ll wait awhile this time.
Posted on Oct 31, 2007 under opensolaris, zfs |
Chenbro has announced a new mini-itx home server/NAS chassis that would make for a super small, super quiet OpenSolaris ZFS storage server. Couple this with an MSI Industrial 945GM1 Core 2 Duo Mainboard, 4GB of memory, a 2.5″ system disk, four 750GB or 1TB data drives, and a cheap four port PCI SATA card, maybe an SD card or two for the slog, and you’ll have yourself a mini-thumper. The only con for me is the single system disk, although there are creative solutions around that.
