mhddfs: combine filesystems

diskThink of a situation where you have multiple disks. Each time you want to store large files on any of them (or you have a growing collection) you may need to check which disk has enough space. It’s a question of keeping your stuff organized too. You may want to use different volumes for Classical, Pop and Rock. What if the volume with Classical music doesn’t have any space for the next Beethoven collection you received?

RAID could be a solution. However, RAID 0 will stripe data across multiple disks and may not be able to recover any data if one of the disks fail. RAID 1 solves this problem but you need to have one mirror disk for each disk and the available space per mirror set is only equal to the capacity of the smallest disk you got. Last but not the least, RAID configuration is tricky.

mhddfs could rescue you from the situation. It is a simple FUSE filesystem module to combine multiple disks. The utility is a cmdline one and is agnostic of the filesystem on each volume as long as the OS understands them.

mhddfs creates a virtual drive having space equal to the sum of all the participating volumes. It has a mlimit option (default 4GB) which determines whether the disk is low on space. When a new file is added, mhddfs copies it to the first participating disk if it has enough free space. If mlimit is already reached it tries the second one and so on. If all the disks have reached mlimit the file is copied to the disk having maximum free space.

If a file copy fails midway due to some reason, mlimit tries to copy it (including already written data) once again to the disk having maximum free space. However, you don’t need to bother because mhddfs gives you a single virtual volume and takes care of all these internally.

mhddfs does not span the data of a single file between multiple disks and hence it’s easy to “disconnect” the disks. You can copy all the data into a single large enough physical volume or you can unmount the mhddfs mountpoint and get back your data split among the disks.

Installation

To install mhddfs on Ubuntu, run:

$ sudo apt-get install mhddfs

Usage

We will start with 3 physical volumes:

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
...
/dev/sda1              80G   50G   30G  63% /mnt/hdd1
/dev/sdb1              40G   35G    5G  88% /mnt/hdd2
/dev/sdc1              60G   10G   50G  17% /mnt/hdd3

To create a large virtual volume from multiple smaller physical volumes, run:

$ sudo mkdir /mnt/virtual
$ sudo mhddfs /mnt/hdd1,/mnt/hdd2,/mnt/hdd3 /mnt/virtual -o allow_other

The allow_other option allows non-root users to read and write the virtual disk.

Check the status of disks:

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
...
/dev/sda1              80G   50G   30G  63% /mnt/hdd1
/dev/sdb1              40G   35G    5G  88% /mnt/hdd2
/dev/sdc1              60G   10G   50G  17% /mnt/hdd3
mhddfs                180G   95G   85G  53% /mnt/virtual

To make this arrangement permanent, add the following to /etc/fstab:

mhddfs#/mnt/hdd1,/mnt/hdd2,/mnt/hdd3 /mnt/virtual fuse defaults,allow_other 0 0

To disconnect the volumes, run:

$ sudo umount /mnt/virtual

Man page: mhddfs

2 thoughts on “mhddfs: combine filesystems”

Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s