Everyone has probably heard about Suns ZFS filesystem, but if you haven’t here is some basic info:

    ZFS presents a pooled storage model that completely eliminates the concept of volumes and the associated problems of partitions, provisioning….

    …the on-disk state is always valid. There is no need to fsck a ZFS file system, ever. Every block is checksummed to prevent silent data corruption, and the data is self-healing in replicated (mirrored or RAID) configurations. If one copy is damaged, ZFS detects it and uses another copy to repair it.

    ZFS loves cheap disks…But cheap disks can fail, so ZFS provides disk scrubbing. Like ECC memory scrubbing, the idea is to read all data to detect latent errors while they’re still correctable. A scrub traverses the entire storage pool to read every copy of every block, validate it against its 256-bit checksum, and repair it if necessary. All this happens while the storage pool is live and in use.

[ad]

Basically, ZFS makes obsolete a number of other technologies we already use daily, from current journaling filesystems, redundant (expensive) RAID hardware, expensive enterprise grade disks, and complicated data integrity protection methods. With one open source software package, Sun has brought some of the best benefits all these things have to offer to anyone who can run the software. For some quick background, Sun already uses ZFS in production servers, FreeBSD includes it in their latest release, Apple includes a read only implementation of it in OS X 10.5 Leopard, and they are planning to include it in the next release of OS X 10.6 Snow Leopard Server.

However Apple also makes available a developer alpha/beta of ZFS for currently shipping OS X systems at http://zfs.macosforge.org/. A word of caution though, this is without question preview quality code, and although geeks like me are already testing and indeed using it, the current release ZFS-119 is known to cause what i would consider serious problems for day to day use, including full kernel panics.

Due to the way the original codebase from Sun is licensed and due to some patent issues, Linux developers have been hesitant to use the upstream ZFS code from Sun, which is under the CDDL license (incompatible with the GPL), or develop a new implementation from the ground up for the Linux kernel. While we won’t cover such details here, hope is not lost: there is an implementation of ZFS for Linux in the form of a FUSE filesystem, and it works quite well, far better than expected.

I have been using ZFS on OS X for a few weeks on my long term application and document storage drive, which is a single disk (though i have multiple copies of the files on it), and it has been fast and reliable so far (unless you yank the usb cable, in which case the kernel panics on OS X). Of particular note, one of the design goals of ZFS is to ensure that the integrity of data on disk is maintained, on a redundant storage pool ZFS will actually find and repair incorrect data using the checksums mentioned in Suns documentation, but on a single disk it will also ensure that data corruption on a single disk does not go unnoticed, a major problem for other filesystems as any sysadmin can tell you. Because of this, one of ZFS first actions in winning my support was to find out for me that one of my newest disks is actually going bad. It is only 7 months old but it is already spitting out bad data when that data is being read back by the system. Similar stories have been told around the web from other ZFS users, it seems to be quite effective in finding bad disks.

Check back soon for part 2: ZFS on OS X currently being written up