This how-to documents how to setup ZFS and install FreeBSD 9.1 into it. ZFS is a great modern file system (invented by SUN Microsystems) to use under FreeBSD. It is very fast and will give you the same features available in a modern SAN (mirrors, raid, snapshots, clones, and pooled storage just to name a few).
 

 

In this document I am building two ZFS pools, one across two drives (zroot) in a mirror configuration and the other (zlocal) that is a RAID setup across 3 drives. The system will be setup to boot from the mirror pool where most of the system resides and the RAID drives will be reserved for stuff under /usr/local. It aligns the ZFS file systems to 4K clusters for newer drives and also does not use ZFS legacy mountpoints. The plan is to have two SSD drives that contain the boot and system files for speed with the RAID drives using some large SATA disks for storage.
 

The first thing to do is to boot from a FreeBSD 9.1 DVD or memstick. Select the “Live CD” option once the system comes up. Login as “root”, there is no password.
 

Partitioning the disk. 

# gpart create -s gpt ad0
# gpart create -s gpt ad1
# gpart create -s gpt ad2
# gpart create -s gpt ad3
# gpart create -s gpt ad4 

Create the boot, swap, and zfs partitions on the two mirror drives. 

# gpart add -b 34 -s 64k -t freebsd-boot ad0
# gpart add -s 4G -t freebsd-swap -l swap0 ad0
# gpart add -t freebsd-zfs -l disk0 ad0 

# gpart add -b 34 -s 64k -t freebsd-boot ad1
# gpart add -s 4G -t freebsd-swap -l swap1 ad1
# gpart add -t freebsd-zfs -l disk1 ad1 

Create zfs patrition on the three RAID drives. 

# gpart add -t freebsd-zfs -l disk2 ad2
# gpart add -t freebsd-zfs -l disk3 ad3
# gpart add -t freebsd-zfs -l disk4 ad4 

Install the Protected MBR and gptzfsboot loader on the mirror drives. 

# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad0
# gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ad1 

Load ZFS Kernel module 

# kldload /boot/kernel/opensolaris.ko
# kldload /boot/kernel/zfs.ko 

Align the disks to 4K clusters. 

# gnop create –S 4096 /dev/gpt/disk0
# gnop create –S 4096 /dev/gpt/disk1
# gnop create –S 4096 /dev/gpt/disk2
# gnop create –S 4096 /dev/gpt/disk3
# gnop create –S 4096 /dev/gpt/disk4 

Setup the ZFS mirror pool (zroot) on the two mirror drives. 

# zpool create -m / –o altroot=/mnt –o cachefile=/var/tmp/zpool.cache zroot mirror /dev/gpt/disk0.nop /dev/gpt/disk1.nop
# zpool export zroot
# gnop destroy /dev/gpt/disk0.nop
# gnop destroy /dev/gpt/disk1.nop
# zpool import –o altroot=/mnt –o cachefile=/var/tmp/zpool.cache zroot 

Create the /tmp file systems 

# zfs create –o compression=on –o exec=on –o setuid=off zroot/tmp
# chmod 1777 /mnt/tmp
 
Create the /usr,/var and /home file systems

# zfs create zroot/usr
# zfs create zroot/usr/home
# cd /mnt; ln –s usr/home home
# zfs create zroot/var 

Setup the ZFS RAID pool (zlocal) on the three extra drives.

# mkdir –p /mnt/usr/local
# zpool create –o altroot=/mnt/usr/local –o cachefile=/var/tmp/zpool.cache zlocal raidz1 /dev/gpt/disk2.nop /dev/gpt/disk3.nop /dev/gpt/disk4.nop
# zpool export zlocal
# gnop destroy /dev/gpt/disk2.nop
# gnop destroy /dev/gpt/disk3.nop
# gnop destroy /dev/gpt/disk4.nop
# zpool import –o altroot=/mnt/usr/local –o cachefile=/var/tmp/zpool.cache zlocal 
 

Setup some final ZFS settings for the boot drives and the checksum algorithm to use.

# zpool set bootfs=zroot zroot
# zfs set checksum=fletcher4 zroot
# zfs set checksum=fletcher4 zlocal


 Create the rest of the file systems


Create the ports file systems 

# zfs create –o compression=lzjb –o setuid=off zroot/usr/ports
# zfs create –o compression=off -o setuid=off –o exec=off zroot/usr/ports/distfiles
# zfs create –o compression=off -o setuid=off –o exec=off zroot/usr/ports/packages
 

Create the system src tree file system. 

# zfs create –o compression=lzjb –o setuid=off –o exec=off zroot/usr/src
 

Create the rest of the file file systems (You can customize this to your needs). 

# zfs create –o compression=lzjb –o setuid=off –o exec=off zroot/var/crash
# zfs create –o compression=off –o setuid=off –o exec=off zroot/var/db
# zfs create –o compression=lzjb –o setuid=off –o exec=on zroot/var/db/pkg
# zfs create –o compression=off –o setuid=off –o exec=off zroot/var/empty
# zfs create –o compression=lzjb –o setuid=off –o exec=off zroot/var/log
# zfs create –o compression=gzip –o setuid=off –o exec=off zroot/var/mail
# zfs create –o compression=off –o setuid=off –o exec=off zroot/var/run
# zfs create –o compression=lzjb –o setuid=off –o exec=on zroot/var/tmp
# chmod 1777 /mnt/var/tmp 

Create the /usr/local file system on the zlocal disks 

# zfs create –o compression=lzjb –o setuid=off –o exec=off zlocal/src 
  

Install FreeBSD onto the new ZFS drives

# sh
# cd /usr/freebsd-dist
# export DESTDIR=/mnt
# for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz games.txz; do (cat $file | tar –-unlink –xpJf - -C ${DESTDIR:-/}); done 

Make /var/empty readable and copy over the ZFS cache file 

# zfs set readonly=on zroot/var/empty
# cp /var/tmp/zpool.cache /mnt/boot/zfs/zpool.cache
 

Create a basic /etc/rc.conf 

# echo ‘zfs_enable=”YES”’ > /mnt/etc/rc.conf
# echo ‘hostname=”beastie.mydomain.local”’ >> /mnt/etc/rc.conf
# ‘ifconfig_em0=”DHCP”’ >> /mnt/etc/rc.conf
 

Create the basic loader.conf 

# echo ‘zfs_load=”YES”’ > /mnt/boot/loader.conf
# echo ‘vfs.root.mountfrom=”zfs:zroot”’ >> /mnt/boot/loader.conf
 

Get the regular nightly status updates. 

# echo 'daily_status_zfs_enable="YES"' >> /mnt/etc/periodic.conf



Finish up the Install
 

Create /etc/fstab to use both swap partitions and support for mounting a CD. 

# cat << EOF > /mnt/etc/fstab
# Device          Mountpoint        FStype      Options     Dump  Pass#
/dev/gpt/swap0    none              swap        sw          0     0
/dev/gpt/swap1    none              swap        sw          0     0
/dev/cd0          /cdrom            cd9660      ro.noauto   0     0
EOF
 

Create the CDrom mount point and then unmount the ZFS file systems. 

# mkdir /mnt/cdrom
# zfs umount –a
 

Change the mountpoints for the zroot and zlocal pools 

# zfs set mountpoint=/ zroot
# zfs set mountpoint=/tmp zroot/tmp
# zfs set mountpoint=/usr zroot/usr
# zfs set mountpoint=/var zroot/var
# zfs set mountpoint=/usr/local zlocal 


Reboot and remove the CD or memstick. It should boot from ZFS on the hard drives.
 

Final Touchups 

One the system reboots into the newly install system. Login and perform the following.
 

Change root’s password 

# passwd 

Setup the local timezone

# tzsetup 

Create the mail aliases file 

# cd /etc/mail
# make aliases 

Customize your new system to your needs. It should be ready to go. 
  

Common ZFS usage and trouble shooting commads.
 

Snapshots are a way to create a quick backup of a ZFS file system. You can then do some work and restore the file system back to the time the snapshot was taken if needed. Snapshots grow over time. This means that taking a snapshot of a volume takes no time at all and doesn’t use any space initially but as the file system gets used after the snapshot and the volumes data diverges farther from what it was at the time of the snapshot, the snapshot size increases. This is why snapshots are not a good solution for long term backups but are great for a quick backup and restore. Snapshots work at the file system (ZFS volume) level so there is no easy way to restore a single file from a snapshot.   Restoring the snapshot restores everything in that file system back to the point the snapshot was taken.
 

In the example below, assume we have a ZFS pool called zlocal and within that pool we have created a volume called zlocal/src.
 

Creating a snapshot of the src file system. 

# zfs rollback zlocal/src@2012-07-28 

Note: The information after the @ sign gives you the snapshot identifier. I usually use the current date so I know when that snapshot was taken.
 

Restoring back to a snapshot. 

# zfs rollback zlocal/src@2012-07-28 

List the available snapshots for the src file system (note. The zlocal/src file system is mounted on /usr/local/src) 

# ls –al /usr/local/src/.zfs/snapshot

Total 2
dr-xr-xr-x 2 root   wheel 2 Jul 28 15:12 .
dr-xr-xr-x 2 root   wheel 2 Jul 27 14:51 ..
dr-xr-xr-x 2 root   wheel 2 Jul 28 15:11 2012-07-28

 

Deleting a snapshot 

# zfs destroy zlocal/src@2012-07-28
 

Recovering from a failed from (RAID-Z Volume). 

To check in the ZFS pool are healthy 

# zpool status –x 

For more information on the ZFS pools 

# zpool status 

You can used the existing checksum information within ZFS to verify the data on a ZFS file system. This runs in the background. Note: scrub sounds bad but it is safe to run. 

# zpool scrub zlocal 

You can check the results of the verify with the status command. 

# zpool status zlocal 

A drive can be taken off line by the administrator with the following command. 

# zpool offline zlocal da3 

The drive can then be replaced and put back online with. 

# zpool replace zlocal da3

 

 

Replacing a mirror drive (one of the boot drives). 

Lets assume that da1 (/dev/gpt/disk1 is the label on that drive) is the drive that failed.
 

Take the drive offline. 

# zpool offline zroot /dev/gpt/disk1 

Remove the drive and replace with the new one. Reboot. 

Create the partitions on the new drive and make it bootable. 

# gpart create –s gpt da1
# gpart add –b 34 –s 64k –t freebsd-boot da1
# gpart add –s 3G –t freebsd-swap –l swap1 da1
# gpart add –t freebsd-zfs –l disk1 da1 

# gpart bootcode –b /boot/pmbr –p /boot/gptzfsboot –i 1 da1 

Aligning to 4K clusters is not required on the new drive as the alignment is on a ZFS pool basis and the 4K alignment is already stored in the metadata for the ZFS pool.

You can check the alignment is correct with the following command. 

# zdb zroot | grep ashift 

The above command should return 12 for a pool that is 4K aligned. 

Replace the drive and add it back into the ZFS pool. 

# zpool replace zroot /dev/gpt/disk1
  

Replacing a RAID-Z (zlocal) drive (not a main boot drive).
 

Lets assume that da3 (/dev/gpt/disk3 is the label on that drive) is the drive that failed. 

Take the drive offline. 

# zpool offline zlocal /dev/gpt/disk3 

Remove the drive and replace with the new one. Reboot. 

Create the partition on the new drive. 

# gpart create –s gpt da3
# gpart add –t freebsd-zfs –l disk3 da3 

Aligning to 4K clusters is not required on the new drive as the alignment is on a ZFS pool basis and the 4K alignment is already stored in the metadata for the ZFS pool.

You can check the alignment is correct with the following command. 

# zdb zlocal | grep ashift 

The above command should return 12 for a pool that is 4K aligned. 

Replace the drive and add it back into the ZFS pool. 

# zpool replace zlocal /dev/gpt/disk3