I am skipping the SmartOS introduction because Magnus Hedemark has us covered with Discovering SmartOS. Instead, I’ll dive straight in and tell you that it’s great as an all-in-one storage and virtualization server, despite only being branded as the latter. Maybe you’ve been using ESXi, Nexenta CP/NexentaStor, or OpenIndiana to fill this role (or even a combination of both on separate machines), but I assure you that it’s possible to have your cake and eat it too. The cake is not a lie!

Once you become familiar with the SmartOS way, you’ll discover that it wants you to start from scratch with clean disks. During the initial configuration, it will prompt you for the disks to use and create a pool named ‘zones’ for storing persistent data on. If you’re like me, you already have an existing data pool from a previous OS install and would like to serve up some zones and KVM instances along with your existing ZFS filesystems. Since the installer doesn’t have provisions for dealing with this scenario we need to improvise a little bit.

You will need a few things before you begin:

  • A SmartOS bootable USB key
  • Output from ‘zpool status’ so you know the drive names and which pool they’re currently in
  • Your existing system disks/OS or an OpenIndiana LiveCD
  • A spare drive SmartOS can temporarily use (all data on this drive will be lost)
  • Good backups of all your data, just in case

The basic steps to accomplish the magic are as follows:

  1. Boot into single user mode from your existing OS
  2. Rename any existing ZFS dataset that might be called ‘zones’:
	# zfs rename data/zones data/old_zones
	# zfs set mountpoint=/old_zones data/old_zones
  1. Export your old pool (mine was called data and is used as an example):
	# zpool export data
  1. Boot using the USB key, follow the prompts, and make sure you install onto the spare disk and not any of your existing data disks (you have that output from ‘zpool status’, right?). After the install, SmartOS should boot, your old data pool should be imported at the mount point specified in step 2, and your spare disk will have a ‘zones’ pool with the following new datasets:
	zones/config
	zones/cores
	zones/opt
	zones/swap
	zones/usbkey
	zones/var
  1. Prepare to transfer the new datasets to your old pool:
	# zfs snapshot -r zones@migrate
	# for dset in "config cores opt swap usbkey var"; do zfs send -R zones/${dset}@migrate | zfs recv data/${dset}; done
  1. Reboot into your previous OS (single-user mode) or a live CD so you can do an Indiana Jones style pool swap. This is necessary because you cannot export the new ‘zones’ pool while SmartOS is booted, and you need to be able to import your old pool as ‘zones’ to effectively rename it. With this done, make the swap:
	# zpool import data zones

You will now technically have two pools called ‘zones’. Chances are good you will not be able to import the one that SmartOS created because it uses a newer ZFS pool version than what is available in other distros. If you don’t intend to use the spare drive in your system, you can power down, remove it, and boot back into SmartOS now. If you’d like to use the spare drive then follow the last step.

  1. Reboot back into SmartOS. When you do, you’ll be prompted to go through configuration again as a result of there being two ‘zones’ pools. Answer no to skip configuration and then you’ll be presented with a login prompt. Login with ‘root’ and the default single-user root password for your SmartOS version. These are text files found in the https://download.joyent.com/pub/iso/ directory, prefixed with SINGLE_USER_ROOT_PASSWD.

After you’re logged in, typing ‘zpool import’ should present you with two ‘zones’ pools. You should proceed to import your old data pool via its listed pool id, e.g:

	# zpool import -f 12736277524734292156

Since you’re planning on using your spare drive, you can either go ahead and add it to the zones pool (as maybe a cache, log, or hot spare), or import it as a separate pool. To do the latter, import via the pool id and specify a new pool name, e.g:

	# zpool import -f 1265591459793332791 sparepool

In any case, you’ll want to make sure that the zones pool on the spare drive has been renamed or destroyed so it doesn’t interfere with your newly imported zones pool. Reboot again and if all goes well, SmartOS will come up configured and you’re good to go! You may now want to cleanup the @migrate snapshots. When you’re confident that you’ve made the right choice with SmartOS and don’t want to go back to your old setup, running ‘zpool upgrade’ is advised so you can take advantage of newer ZFS pool features.

I have personally used this procedure on two production systems and it works well (this blog is running on one of them). Both of these systems serve up ZFS filesystems via NFS to zones and KVM instances running on the same physical host, as well as other external hosts.