The procedure for fixing this is:

Find the corresponding virtual memory hex address on the left for the stuck log device on the right.

# echo '::spa -v | mdb -k'
...
ffffff0946273d40 HEALTHY - /dev/dsk/c4t5000CCA000342E60d0s0
ffffff0946274380 HEALTHY - /dev/dsk/c7d0s0
- - - cache
ffffff094b069680 HEALTHY - /dev/dsk/c7d0s1


Find the corresponding virtual memory hex address on the left for the field vdev_stat.vs_alloc.

# echo 'ffffff0946273d40::print -a vdev_t vdev_stat' | mdb -k
vdev_stat = {
ffffff0946273f70 vdev_stat.vs_timestamp = 0x52bcba2a6c
ffffff0946273f78 vdev_stat.vs_state = 0
ffffff0946273f80 vdev_stat.vs_aux = 0
ffffff0946273f88 vdev_stat.vs_alloc = 0xfffffffffffe0000
ffffff0946273f90 vdev_stat.vs_space = 0x222c000000
ffffff0946273f98 vdev_stat.vs_dspace = 0x222c000000
ffffff0946273fa0 vdev_stat.vs_rsize = 0
ffffff0946273fa8 vdev_stat.vs_ops = [ 0x5, 0x1c, 0x20b, 0, 0, 0x92 ]
ffffff0946273fd8 vdev_stat.vs_bytes = [ 0, 0x12a000, 0x640000, 0, 0, 0 ]
ffffff0946274008 vdev_stat.vs_read_errors = 0
ffffff0946274010 vdev_stat.vs_write_errors = 0
ffffff0946274018 vdev_stat.vs_checksum_errors = 0
ffffff0946274020 vdev_stat.vs_self_healed = 0
ffffff0946274028 vdev_stat.vs_scan_removing = 0x1
ffffff0946274030 vdev_stat.vs_scan_processed = 0
}


Change the value. Depending on the size of the current value, you may need to use /Z0 to zero the field.

# echo 'ffffff0946273f88/Z0' | mdb -kw
0xffffff0946273f88: 0xffffffff00000000 = 0x0


Verify that ‘zpool iostat -v <pool>’ shows the alloc column for the stuck log device is now 0 and then re-issue the zpool remove command.

# zpool iostat -v data
capacity   operations bandwidth
pool                      alloc  free  read write  read write
------------------------- ----- ----- ----- ----- ----- -----
...
c4t5000CCA000342E60d0         0  137G     0     0     0     0
c7d0s0                    16.0E 9.94G     0     0     0     0
cache                         -     -     -     -     -     -
c7d0s1                    64.9G 7.78M    17    59  394K 7.07M
------------------------- ----- ----- ----- ----- ----- -----

# zpool remove data c4t5000CCA000342E60d0


I experienced this problem on NexentaCore 3.1 which shares the same kernel bits with NexentaStor 3.x. Someone apparently filed a bug with Oracle (CR 7000154) but the information is locked away behind their paid support portal for those that don’t have access. I haven’t looked recently but at the time there wasn’t an Illumos bug filed and it’s still not fixed. Hopefully the above information will help others experiencing this problem, however, please note that I am not responsible for any damage this might do to your data. Please make sure you have things backed up, and be aware that inputting anything to mdb in this manner can cause a kernel panic on a running system!

In order to make this production worthy, it needs to integrate well with my existing LDAP and NFS environment. Snow Leopard Server comes with its own built-in Open Directory based on OpenLDAP, but my LDAP instances are based on OpenDS. In researching what would be necessary to make SLS talk to OpenDS, I came across Brent Kearney’s excellent Integrating Leopard Server With UNIX LDAP blog posts. His work is similar to a post by Rajeev Karamchedu that covers this for Tiger. In both cases they are using Sun’s Directory Server product which is similar to but different than OpenDS.

Suffice it to say that I’m not going to reproduce Brent’s work but I will document changes that I’ve had to make in order to make it work on Snow Leopard. The first of such changes was converting the apple.schema file to something that OpenDS would be happy with. This was accomplished by using Ludovic Poitou’s script for converting OpenLDAP schemas to OpenDS LDIF format. The resulting file still had a few errors that needed to be manually corrected but they were trivial. A working version of the file can be downloaded here. Note that the original apple.schema relies on some parts of the Samba schema. I modified mine to use 50-samba.ldif which came from the OpenDS wiki.

With these two extended schema files in place, all of the special Apple LDAP attributes are now available to me. The files should also work with OpenDJ which I plan on migrating to in the near future.

The new box came together after I scored a good deal on several 3U Chenbro chassis that included rails and a redundant PSU. The full components list includes:

• Chenbro RM31212 3U Rackmount Chassis
• Intel Xeon E5620 Westmere 2.4GHz Quad-Core CPU (x2)
• Supermicro X8DTH-6F Motherboard
• Crucial 4GB DDR3 PC3-8500 DIMM (x8)
• OCZ Vertex 2 40GB SATAII SSDs (x2 for mirrored zfs system pool)
• Intel X25-M 80GB SATAII SSD (L2ARC read cache)
• Hitachi 7K2000 SATAII Hard Drives (x6 for raidz2 mirrored zfs data pool)

The Supermicro motherboard has dual SAS2 ports onboard that posed a small challenge for the Chenbro case, but it was nothing a little bit of dremel tool action couldn’t solve. Other than that everything went together well and I’m really happy with how it turned out. I have plenty of room for expansion with six additional hot-swap bays, four additional memory slots, and seven PCI Express slots. This bad boy is running NCP 3.0.1 along with most of the newer packages I’ve been backporting. It should be able to handle quite a large number of zones and VirtualBox VMs among other things.

Here are a few pictures I took after everything was assembled:

These packages are pulled in from a variety of sources but most of them come from Debian (unstable). Note that I make no guarantees of any kind so use these packages at your own risk. I use them in production for machines at work and elsewhere but it doesn’t mean you should. Feel free to leave feedback though if you run into any problems.

To add the repository, edit /etc/apt/sources.list and include:

deb http://apt.zpool.org/ hardy-unstable main


Fortunately there is a simple, relatively inexpensive way to use SSD drives with your Thumper. Here’s what you need:

You’ll also want a spare X4500 drive bracket unless you’re willing to use a bracket from the drive(s) you have to pull out. Insert your Intel X25-E SSD into the Icy Dock, attach the Icy Dock to the X4500 bracket like any 3.5″ drive, and insert the result into your Thumper. Voila. The Icy Dock bay is well constructed despite being plastic (I would prefer it was made out of aluminum). Also, $20 for a hunk of plastic and a converter isn’t ideal, but those of you that would suggest duct tape have no sense of aesthetics.

In my case, I swapped out two 1TB drives for two 32GB Intel X25-E SSDs. I then partitioned the SSDs into two slices, one for mirrored system disks and the other for ZIL. They show up and work just fine:

platform = Sun Fire X4500

Device    Serial        Vendor   Model             Rev  Temperature
------    ------        ------   -----             ---- -----------
c0t0d0p0  E002PAJTBX4E  ATA      HITACHI HUA7210S  A90A 28 C (82 F)
c0t1d0p0  E002PAJTBWTE  ATA      HITACHI HUA7210S  A90A 30 C (86 F)
c0t2d0p0  F002PAJSY2AF  ATA      HITACHI HUA7210S  A90A 33 C (91 F)
c0t3d0p0  F002PAJSV5BF  ATA      HITACHI HUA7210S  A90A 35 C (95 F)
c0t4d0p0  E002PAJTBX1E  ATA      HITACHI HUA7210S  A90A 28 C (82 F)
c0t5d0p0  E002PAJTBWME  ATA      HITACHI HUA7210S  A90A 32 C (89 F)
c0t6d0p0  F002PAJS156F  ATA      HITACHI HUA7210S  A90A 34 C (93 F)
c0t7d0p0  F002PAJSW14F  ATA      HITACHI HUA7210S  A90A 35 C (95 F)
c1t0d0p0  E002PAJT4VLE  ATA      HITACHI HUA7210S  A90A 29 C (84 F)
c1t1d0p0  E002PAJTBWWE  ATA      HITACHI HUA7210S  A90A 32 C (89 F)
c1t2d0p0  F002PAJSZP2F  ATA      HITACHI HUA7210S  A90A 33 C (91 F)
c1t3d0p0  E002PAJT5BUE  ATA      HITACHI HUA7210S  A90A 35 C (95 F)
c1t4d0p0  E002PAJTBX7E  ATA      HITACHI HUA7210S  A90A 28 C (82 F)
c1t5d0p0  E002PAJTBX3E  ATA      HITACHI HUA7210S  A90A 31 C (87 F)
c1t6d0p0  F002PAJS1DEF  ATA      HITACHI HUA7210S  A90A 33 C (91 F)
c1t7d0p0  E002PAJT541E  ATA      HITACHI HUA7210S  A90A 35 C (95 F)
c5t0d0p0  E002PAJTA8WE  ATA      HITACHI HUA7210S  A90A 28 C (82 F)
c5t1d0p0  E002PAJTAJ0E  ATA      HITACHI HUA7210S  A90A 31 C (87 F)
c5t2d0p0  E002PAJT50KE  ATA      HITACHI HUA7210S  A90A 33 C (91 F)
c5t3d0p0  E002PAJTB9LE  ATA      HITACHI HUA7210S  A90A 35 C (95 F)
c5t4d0p0  E002PAJTAGME  ATA      HITACHI HUA7210S  A90A 28 C (82 F)
c5t5d0p0  E002PAJT58JE  ATA      HITACHI HUA7210S  A90A 31 C (87 F)
c5t6d0p0  E002PAJT52TE  ATA      HITACHI HUA7210S  A90A 34 C (93 F)
c5t7d0p0  E002PAJT4UNE  ATA      HITACHI HUA7210S  A90A 35 C (95 F)
c6t0d0p0  140067032HGN  ATA      SSDSA2SH032G1GN   8850 255 C (491 F)
c6t1d0p0  E002PAJTAJ6E  ATA      HITACHI HUA7210S  A90A 30 C (86 F)
c6t2d0p0  F002PAJSZNEF  ATA      HITACHI HUA7210S  A90A 33 C (91 F)
c6t3d0p0  E002PAJTBH7E  ATA      HITACHI HUA7210S  A90A 35 C (95 F)
c6t4d0p0  1400CB032HGN  ATA      SSDSA2SH032G1GN   8850 255 C (491 F)
c6t5d0p0  E002PAJT120E  ATA      HITACHI HUA7210S  A90A 30 C (86 F)
c6t6d0p0  F002PAJS79AF  ATA      HITACHI HUA7210S  A90A 32 C (89 F)
c6t7d0p0  E002PAJT5UZE  ATA      HITACHI HUA7210S  A90A 34 C (93 F)
c7t0d0p0  E002PAJTBVJE  ATA      HITACHI HUA7210S  A90A 29 C (84 F)
c7t1d0p0  E002PAJTAJ3E  ATA      HITACHI HUA7210S  A90A 31 C (87 F)
c7t2d0p0  E002PAJT53ME  ATA      HITACHI HUA7210S  A90A 33 C (91 F)
c7t3d0p0  F002PAJSZNDF  ATA      HITACHI HUA7210S  A90A 34 C (93 F)
c7t4d0p0  E002PAJTBL8E  ATA      HITACHI HUA7210S  A90A 27 C (80 F)
c7t5d0p0  E002PAJTAK4E  ATA      HITACHI HUA7210S  A90A 31 C (87 F)
c7t6d0p0  F002PAJS56AF  ATA      HITACHI HUA7210S  A90A 32 C (89 F)
c7t7d0p0  F002PAJSAW1F  ATA      HITACHI HUA7210S  A90A 35 C (95 F)
c8t0d0p0  E002PAJTAHTE  ATA      HITACHI HUA7210S  A90A 28 C (82 F)
c8t1d0p0  E002PAJSRR8E  ATA      HITACHI HUA7210S  A90A 31 C (87 F)
c8t2d0p0  F002PAJT7YAF  ATA      HITACHI HUA7210S  A90A 33 C (91 F)
c8t3d0p0  E002PAJT4W9E  ATA      HITACHI HUA7210S  A90A 35 C (95 F)
c8t4d0p0  E002PAJT583E  ATA      HITACHI HUA7210S  A90A 28 C (82 F)
c8t5d0p0  E002PAJTAG8E  ATA      HITACHI HUA7210S  A90A 31 C (87 F)
c8t6d0p0  E002PAJTBATE  ATA      HITACHI HUA7210S  A90A 33 C (91 F)
c8t7d0p0  E002PAJT52UE  ATA      HITACHI HUA7210S  A90A 34 C (93 F)

Updated: 2010/04/02

The high temperature shown for the SSDs is simply due to Sun’s hd utility not comprehending the return values properly. This is probably due to non-Sun firmware or lack of SSD specific awareness but it does not affect operation. There have been other reports of problems with FMA marking the drives as faulty (which may or may not bother you), but the most recent Intel firmware seems to have resolved this. I have been running these SSDs in two X4500s for nearly a year with no problems.

Here’s an example of what happens:


# apt-clone -v dist-upgrade
This operation will upgrade your system using ZFS capabilities. Proceed ? (y/n) y

apt-get -V -y install dpkg apt
Reading package lists... Done
Building dependency tree
Reading state information... Done
dpkg is already the newest version.
apt is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 16 not upgraded.
rmdir /tmp/upgrade-attempt.21883 2>/dev/null
mkdir /tmp/upgrade-attempt.21883
mount -F zfs syspool/rootfs-nmu-002 /tmp/upgrade-attempt.21883 2>/dev/null
chroot /tmp/upgrade-attempt.21883 mount /proc 2>/dev/null
mount -F lofs -O /etc/mnttab /tmp/upgrade-attempt.21883/etc/mnttab 2>/dev/null
mount -F lofs -O /devices /tmp/upgrade-attempt.21883/devices 2>/dev/null
APT_CLONE_ENV=1 chroot /tmp/upgrade-attempt.21883 apt-get -V -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
man-db (2.5.2-2nexenta1 => 2.5.2-2nexenta2)
sunwcakr (5.11.104-2 => 5.11.104-3)
sunwckr (5.11.104-4 => 5.11.104-5)
sunwcsl (5.11.104-5 => 5.11.104-6)
sunwemlxs (5.11.104-1 => 5.11.104-2)
sunwiscsidmr (5.11.104-1 => 5.11.104-2)
sunwiscsidmu (5.11.104-1 => 5.11.104-2)
sunwiscsir (5.11.104-5 => 5.11.104-6)
sunwiscsitr (5.11.104-1 => 5.11.104-2)
sunwiscsitu (5.11.104-1 => 5.11.104-2)
sunwkrbu (5.11.104-1 => 5.11.104-2)
sunwstmf (5.11.104-1 => 5.11.104-2)
sunwstmfu (5.11.104-1 => 5.11.104-2)
sunwzfskr (5.11.104-5 => 5.11.104-7)
sunwzfsr (5.11.104-4 => 5.11.104-6)
sunwzfsu (5.11.104-4 => 5.11.104-6)
16 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 20.8MB/38.9MB of archives.
After this operation, 3048kB disk space will be freed.
Do you want to continue [Y/n]? y
Get:1 http://apt.nexenta.org hardy-unstable/main sunwcakr 5.11.104-3 [2462kB]
Get:2 http://apt.nexenta.org hardy-unstable/main sunwckr 5.11.104-5 [14.5MB]
Get:3 http://apt.nexenta.org hardy-unstable/main sunwzfsr 5.11.104-6 [453kB]
Get:4 http://apt.nexenta.org hardy-unstable/main man-db 2.5.2-2nexenta2 [1050kB]
Get:5 http://apt.nexenta.org hardy-unstable/main sunwiscsir 5.11.104-6 [236kB]
Get:6 http://apt.nexenta.org hardy-unstable/main sunwzfskr 5.11.104-7 [851kB]
Get:7 http://apt.nexenta.org hardy-unstable/main sunwzfsu 5.11.104-6 [1253kB]
Fetched 20.8MB in 49s (420kB/s)
nl.1: sh: fatal: relocation error: file /lib/libncurses.so.5: symbol main: referenced symbol not found
E: Sub-process /usr/bin/apt-clone -c exited unexpectedly
E: Failure running script /usr/bin/apt-clone -c
chroot /tmp/upgrade-attempt.21883 umount /proc 2>/dev/null
umount /tmp/upgrade-attempt.21883/etc/mnttab 2>/dev/null
umount /tmp/upgrade-attempt.21883/devices 2>/dev/null
rm -rf /tmp/nl*
umount /tmp/upgrade-attempt.21883 2>/dev/null
rmdir /tmp/upgrade-attempt.21883
Upgrade failed. Would you like to rollback changes now? (y/n)


In order to fix this, you would answer no to the rollback and then execute something like this:


# mount -F zfs syspool/rootfs-nmu-002 /mnt
# chroot /mnt mount /proc
# mount -F lofs -O /etc/mnttab /mnt/etc/mnttab
# mount -F lofs -O /devices /mnt/devices
# chroot /mnt


Now from within the chroot, execute:


# rm -r /tmp/nl*
# APT_CLONE_ENV=1 apt-get -V -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" dist-upgrade
# bootadm update-archive
# exit


This should get all the packages within the clone upgraded properly, update the boot archive, and back you out of the clone. Now you need to install a new grub, unmount the clone, and reboot:


# installgrub /mnt/boot/grub/stage1 /mnt/boot/grub/stage2 /dev/rdsk/
# installgrub /mnt/boot/grub/stage1 /mnt/boot/grub/stage2 /dev/rdsk/ (if you have one)
# chroot /mnt umount /proc
# umount /mnt/devices
# umount /mnt/etc/mnttab
# umount /mnt


Obviously you’ll need to replace with the correct device(s), such as c0d0s0 and c1d0s0. If you don’t know which devices to use, run zpool status syspool and it should tell you. Note above that we did not activate the new clone as the default bootfs. It is best now to reboot, select the new entry from the grub menu, and verify that the clone is working properly. Once you’re booted into the clone, you can run apt-clone -a rootfs-nmu-002 to make it the default.

Hopefully this helps some of you out there having problems. Feel free to drop by ##nexenta on irc.freenode.net if you require more assistance.

The full components list includes:

• Chenbro RM31212B 3U Rackmount Chassis including a 460W Zippy PSU
• Intel Core i7 920 Nehalem 2.66GHz Quad-Core
• Crucial 6GB 240-Pin DDR3 SDRAM (PC3 10600) Triple Channel Memory Kit
• Supermicro MBD-X8SAX-O Intel X58 Server Motherboard
• Seagate Barracuda 7200.9 ST3120813AS 120GB 7200 RPM Sata Hard Drive(x2 system drives)
• Seagate Barracuda 7200.8 ST3250823AS 250GB 7200 RPM Sata Hard Drive (x6 data drives)
• Seagate Barracuda 7200.11 ST31000340AS 1TB 7200 RPM SATA Hard Drive (x6 data drives)
• SYBA SD-SA2PEX-2IR PCI-e SATA II card (x2)
• Supermicro AOC-SAT-MV8 (x1)
• Ancient Matrox PCI video card

platinum.alloy.net serves up audio/video/photos/etc. to two hacked AppleTV boxes, my wife’s iMac, and my Mac Pro via Samba. It also serves up home directories for the Sun Ray thin-client in the kitchen and acts as a backup server for our Macbook Pro laptops.

It has been through a number of incarnations over the years. I picked up the Chenbro case off eBay a year or so ago for around $350 which was a steal, considering they run upwards of $800 new. The motherboard, CPU, memory, and 1TB drives are new as of December 2008 (early Christmas present, yay!). Everything else I’ve had for a couple of years and it’s been the usual organic upgrade process. The Supermicro MBD-X8SAX-O motherboard is so new that I must have been the first one to buy it off Newegg. The price for the new motherboard/cpu/memory dropped about $100 less than a week after I ordered, and I hadn’t even received the parts yet. Typical! I tend not to be so bleeding edge and only upgrade every three years or so, but the timing was right and Nehalem was calling me in my sleep.

I was previously using a Tyan S2865G2NR board with a 2.2GHz AMD Athlon X2 and 4GB of Crucial memory. There was nothing wrong with this other than it didn’t have enough PCI-X or PCI-E slots, and socket 939 is getting a bit long in the tooth. The upshot is that I can get ~$200 out of the old parts on eBay and recoup some of the bleeding edge cost.

Upgrading to the new motherboard was mostly straightforward, although I had to do a manual dist-upgrade to the latest NexentaCP unstable since the OpenSolaris b85 kernel had problems booting. I burned NexentaCore 2.0 Alpha1 and used the recovery grub option to login, mount my syspool, and fix things. I’m in the process of migrating all the data off the six Seagate 250GB drives which are attached to the AOC-SAT-MV8. This card has had a lot of problems with ZFS and has been discontinued by Supermicro in favor of the AOC-SAT2-MV8. Whenever I expand beyond six disks, I’m going to skip straight to the AOC-USAS-L8i using mini-SAS to SATA cables.

So far the new board/cpu/memory appear stable, and it’s been serving up WALL-E repeatedly to our three year old for a few days now. It’s nice to see Hyperthreading giving us what appears as 8 core bling:


# psrinfo -v
Status of virtual processor 0 as of: 12/15/2008 22:56:39
on-line since 12/14/2008 18:22:47.
The i386 processor operates at 2660 MHz,
and has an i387 compatible floating point processor.
Status of virtual processor 1 as of: 12/15/2008 22:56:39
on-line since 12/14/2008 18:22:51.
The i386 processor operates at 2660 MHz,
and has an i387 compatible floating point processor.
Status of virtual processor 2 as of: 12/15/2008 22:56:39
on-line since 12/14/2008 18:22:51.
The i386 processor operates at 2660 MHz,
and has an i387 compatible floating point processor.
Status of virtual processor 3 as of: 12/15/2008 22:56:39
on-line since 12/14/2008 18:22:51.
The i386 processor operates at 2660 MHz,
and has an i387 compatible floating point processor.
Status of virtual processor 4 as of: 12/15/2008 22:56:39
on-line since 12/14/2008 18:22:51.
The i386 processor operates at 2660 MHz,
and has an i387 compatible floating point processor.
Status of virtual processor 5 as of: 12/15/2008 22:56:39
on-line since 12/14/2008 18:22:51.
The i386 processor operates at 2660 MHz,
and has an i387 compatible floating point processor.
Status of virtual processor 6 as of: 12/15/2008 22:56:39
on-line since 12/14/2008 18:22:51.
The i386 processor operates at 2660 MHz,
and has an i387 compatible floating point processor.
Status of virtual processor 7 as of: 12/15/2008 22:56:39
on-line since 12/14/2008 18:22:51.
The i386 processor operates at 2660 MHz,
and has an i387 compatible floating point processor.


I’ll work on benchmarking this home-grown thumper after the New Year. The next upgrade is to install four Intel X25-M SSDs for the ZIL and L2ARC caches. Given the cost I think I’ll wait awhile this time.

My Sun Ray environment today is using Ubuntu Hardy and Flash 10 from the included flashplugin-nonfree package. I’m still using PulseAudio with a workaround by Sebastian Hesselbarth that nicely configures pulseaudio for each user and redirects sound through the appropriate $UTAUDIODEV device. No more libflashsupport dependency. With this in place, users launch the Sound configuration in GNOME and set all options to “PulseAudio Sound Server”, and it just works.

Unfortunately the smurfing effect has returned in Flash 10, having been temporarily fixed at one point in a Flash 9 beta. Someone filed a bug with Adobe recently about this, and it appears it is being investigated. Hopefully Adobe can fix this soon!

In the meantime, I’m making my experimental APT repository available. This is just a collection of packages I’ve had to compile over the last year or so to support work related projects. Some of these packages require manually patching and other changes to get compiled properly and I am remiss in getting these submitted upstream. My next goal is to get as many of these as possible (within the repository guidelines) into the official Nexenta APT repository and actually start maintaining some of them. Anyway, I hope some of these packages prove useful to others.