I ran into a troublesome ZFS bug several months ago where a pool with a log device became “stuck”. The ‘zpool remove’ command would complete but would not remove the device. This was a bad place to be in, because the device was no longer usable, could not be removed, and would most likely prevent the pool from ever being exported and reimported again. Someone else had posted on the zfs-discuss mailing list about the same problem and put me in contact with George Wilson, who in turn put me on the right track to a successful workaround. Log devices exhibiting this behavior need to have a specific field of a data structure zeroed out, and then the remove command will actually finish removing the device.

The procedure for fixing this is:

Find the corresponding virtual memory hex address on the left for the stuck log device on the right.

# echo '::spa -v | mdb -k'
...
ffffff0946273d40 HEALTHY - /dev/dsk/c4t5000CCA000342E60d0s0
ffffff0946274380 HEALTHY - /dev/dsk/c7d0s0
- - - cache
ffffff094b069680 HEALTHY - /dev/dsk/c7d0s1


Find the corresponding virtual memory hex address on the left for the field vdev_stat.vs_alloc.

# echo 'ffffff0946273d40::print -a vdev_t vdev_stat' | mdb -k
vdev_stat = {
ffffff0946273f70 vdev_stat.vs_timestamp = 0x52bcba2a6c
ffffff0946273f78 vdev_stat.vs_state = 0
ffffff0946273f80 vdev_stat.vs_aux = 0
ffffff0946273f88 vdev_stat.vs_alloc = 0xfffffffffffe0000
ffffff0946273f90 vdev_stat.vs_space = 0x222c000000
ffffff0946273f98 vdev_stat.vs_dspace = 0x222c000000
ffffff0946273fa0 vdev_stat.vs_rsize = 0
ffffff0946273fa8 vdev_stat.vs_ops = [ 0x5, 0x1c, 0x20b, 0, 0, 0x92 ]
ffffff0946273fd8 vdev_stat.vs_bytes = [ 0, 0x12a000, 0x640000, 0, 0, 0 ]
ffffff0946274008 vdev_stat.vs_read_errors = 0
ffffff0946274010 vdev_stat.vs_write_errors = 0
ffffff0946274018 vdev_stat.vs_checksum_errors = 0
ffffff0946274020 vdev_stat.vs_self_healed = 0
ffffff0946274028 vdev_stat.vs_scan_removing = 0x1
ffffff0946274030 vdev_stat.vs_scan_processed = 0
}


Change the value. Depending on the size of the current value, you may need to use /Z0 to zero the field.

# echo 'ffffff0946273f88/Z0' | mdb -kw
0xffffff0946273f88: 0xffffffff00000000 = 0x0


Verify that ‘zpool iostat -v <pool>’ shows the alloc column for the stuck log device is now 0 and then re-issue the zpool remove command.

# zpool iostat -v data
capacity   operations bandwidth
pool                      alloc  free  read write  read write
------------------------- ----- ----- ----- ----- ----- -----
...
c4t5000CCA000342E60d0         0  137G     0     0     0     0
c7d0s0                    16.0E 9.94G     0     0     0     0
cache                         -     -     -     -     -     -
c7d0s1                    64.9G 7.78M    17    59  394K 7.07M
------------------------- ----- ----- ----- ----- ----- -----

# zpool remove data c4t5000CCA000342E60d0


I experienced this problem on NexentaCore 3.1 which shares the same kernel bits with NexentaStor 3.x. Someone apparently filed a bug with Oracle (CR 7000154) but the information is locked away behind their paid support portal for those that don’t have access. I haven’t looked recently but at the time there wasn’t an Illumos bug filed and it’s still not fixed. Hopefully the above information will help others experiencing this problem, however, please note that I am not responsible for any damage this might do to your data. Please make sure you have things backed up, and be aware that inputting anything to mdb in this manner can cause a kernel panic on a running system!

I have been running VirtualBox on NCP for nearly a year now both in production and development environments, thanks to some earlier work by Ernst Gill. I’ve finally found the time to update his work for VirtualBox 4.0.6. I have unfortunately not had time to test this thoroughly so I’d appreciate feedback if you find something wrong with it. You can obtain the patch here.

I recently purchased a pair of Apple Xserve boxes for a project at work, with the goal of providing a functional Mac OS X desktop environment via our existing Sun Ray thin-clients. These two systems will be setup using Aqua Connect aka ACTS in a load-balanced, Windows Terminal Server like fashion.

In order to make this production worthy, it needs to integrate well with my existing LDAP and NFS environment. Snow Leopard Server comes with its own built-in Open Directory based on OpenLDAP, but my LDAP instances are based on OpenDS. In researching what would be necessary to make SLS talk to OpenDS, I came across Brent Kearney’s excellent Integrating Leopard Server With UNIX LDAP blog posts. His work is similar to a post by Rajeev Karamchedu that covers this for Tiger. In both cases they are using Sun’s Directory Server product which is similar to but different than OpenDS.

Suffice it to say that I’m not going to reproduce Brent’s work but I will document changes that I’ve had to make in order to make it work on Snow Leopard. The first of such changes was converting the apple.schema file to something that OpenDS would be happy with. This was accomplished by using Ludovic Poitou’s script for converting OpenLDAP schemas to OpenDS LDIF format. The resulting file still had a few errors that needed to be manually corrected but they were trivial. A working version of the file can be downloaded here. Note that the original apple.schema relies on some parts of the Samba schema. I modified mine to use 50-samba.ldif which came from the OpenDS wiki.

With these two extended schema files in place, all of the special Apple LDAP attributes are now available to me. The files should also work with OpenDJ which I plan on migrating to in the near future.

It has been about six years since I last upgraded my co-lo box that hosts a myriad of services and websites for opensource, business, friends, and personal hobbies. Now I have a new machine in place that should last at least another six years. A tremendous amount of thanks goes out to Andrew at NETPLEX for dealing with the inevitable problems that come up when you ship a machine nearly 1000 miles. No matter how prepared you think you are it’s. never. easy. Regardless, the guys at NETPLEX went way beyond the call of duty to help get everything installed and setup so I could proceed with the migration.

The new box came together after I scored a good deal on several 3U Chenbro chassis that included rails and a redundant PSU. The full components list includes:

• Chenbro RM31212 3U Rackmount Chassis
• Intel Xeon E5620 Westmere 2.4GHz Quad-Core CPU (x2)
• Supermicro X8DTH-6F Motherboard
• Crucial 4GB DDR3 PC3-8500 DIMM (x8)
• OCZ Vertex 2 40GB SATAII SSDs (x2 for mirrored zfs system pool)
• Intel X25-M 80GB SATAII SSD (L2ARC read cache)
• Hitachi 7K2000 SATAII Hard Drives (x6 for raidz2 mirrored zfs data pool)

The Supermicro motherboard has dual SAS2 ports onboard that posed a small challenge for the Chenbro case, but it was nothing a little bit of dremel tool action couldn’t solve. Other than that everything went together well and I’m really happy with how it turned out. I have plenty of room for expansion with six additional hot-swap bays, four additional memory slots, and seven PCI Express slots. This bad boy is running NCP 3.0.1 along with most of the newer packages I’ve been backporting. It should be able to handle quite a large number of zones and VirtualBox VMs among other things.

Here are a few pictures I took after everything was assembled:

I have re-established my local Nexenta repository and am making it available to the public. If you are interested in newer versions of Apache, PHP, and Samba compatible with NCP 2.x or 3.x then you’ll want to peruse apt.zpool.org. Some other tidbits include the latest stable BCFG2, a patched gamin package that supports Solaris FEN, and a working php5-imap package.

These packages are pulled in from a variety of sources but most of them come from Debian (unstable). Note that I make no guarantees of any kind so use these packages at your own risk. I use them in production for machines at work and elsewhere but it doesn’t mean you should. Feel free to leave feedback though if you run into any problems.

To add the repository, edit /etc/apt/sources.list and include:

deb http://apt.zpool.org/ hardy-unstable main