LVMify the disk

LVM? What's this all about, anyway?

LVM Linux's Logical Volume Manager. Wikipedia describes the Linux LVM here. Basically, it allows you to take a pool of one or more hard drives and create one or more virtual hard drives out of piecs of the real drives. These can then be resized at run-time, allowing the admin to grow (or sometimes) shrink filesystems while the machine is up and running. This page covers the moving of a hard-partitioned Seagate DockStar running Debian to an LVM-based setup.

Before LVM-ification

Got a backup? You could bork your machine pretty hard if something bad happens. You've been warned.

Here's how the DockStar's USB flash drive looks now with regard to filesystem free space and partitioning:

root@canopus:~# fdisk -l -u /dev/sda

Disk /dev/sda: 4000 MB, 4000317440 bytes
64 heads, 32 sectors/track, 3815 cylinders, total 7813120 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9a198ead

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1050623      524288   83  Linux
/dev/sda2         1050624     2099199      524288   83  Linux
/dev/sda3         2099200     7813119     2856960   8e  Linux LVM
root@canopus:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                  516040    311876    177952  64% /
none                     60492        36     60456   1% /dev
/dev/sda1               516040    311876    177952  64% /
tmpfs                    62864         0     62864   0% /lib/init/rw
tmpfs                    62864         0     62864   0% /dev/shm
tmpfs                    62864         0     62864   0% /tmp
root@canopus:~# cat /etc/fstab
# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/root      /               ext2    noatime,errors=remount-ro 0 1
/dev/sda2      none            swap    sw                0       0
tmpfs          /tmp            tmpfs   defaults          0       0
root@canopus:~#

The current situation

As you can probably tell, the only filesystem is on /dev/sda1 and /dev/sda2 is used for swap space. /dev/sda3 is not used for anything at this point.

The desired end state

I'd like to end up with /dev/sda1 used purely for system configuration and needed-for-boot files. Swap space, /var, /home, /tmp, and /usr will all live in LVM managed block devices. /dev/sda2 will be converted from swap space to a working backup copy of the current root filesystem and used for repairing the real system should the sysadmin suffer a severe bout of brain-damage and break the thing really bad.

Based on lots of use of du, I believe 512Mbytes for swap, 64Mbytes for /home, 128Mbytes for /tmp, 192Mbytes for /usr, and 192Mbytes for /var will be a good target configuration.

Creating an LVM physical volume

Some data structures need to be placed on a physical device that LVM uses. This is done with the pvcreate program. /dev/sda3 is where we'll put the LVM managed logical volumes, so pvcreate is invoked so (in test mode first, to be sure our command line is sane):

root@canopus:~# pvcreate --test --verbose --metadatatype 2 --pvmetadatacopies 2 --zero y /dev/sda3
  Test mode: Metadata will NOT be updated.
    Set up physical volume for "/dev/sda3" with 5713920 available sectors
    Zeroing start of device /dev/sda3
  Physical volume "/dev/sda3" successfully created
    Test mode: Wiping internal cache
    Wiping internal VG cache
root@canopus:~# pvcreate --verbose --metadatatype 2 --pvmetadatacopies 2 --zero y /dev/sda3
    Set up physical volume for "/dev/sda3" with 5713920 available sectors
    Zeroing start of device /dev/sda3
  Physical volume "/dev/sda3" successfully created
root@canopus:~# 

Creating a volume group

Now that there is a physical volume, we can create a volume group. We're specifying that this volume group will not be clustered, is an LVM2 volume group (the tools support the older LVM1 format, too), a physical extent size of 64Mibibytes (this is the size of the pieces that go into any given logical volume. Since the backing store for this volume group is about 3Gibibytes in size, a 64Mibibyte gives us 43 pieces of that to play with), a volume group name of vg00, and that /dev/sda3 is our physical backing device. Again, test mode is used to ensure sanity of command line before really creating the volume group.

root@canopus:~# vgcreate --test --verbose --clustered n --metadatatype 2 --physicalextentsize 64M vg00 /dev/sda3
  Test mode: Metadata will NOT be updated.
    Wiping cache of LVM-capable devices
    Wiping cache of LVM-capable devices
    Adding physical volume '/dev/sda3' to volume group 'vg00'
    Test mode: Skipping archiving of volume group.
    Test mode: Skipping volume group backup.
  Volume group "vg00" successfully created
    Test mode: Wiping internal cache
    Wiping internal VG cache
root@canopus:~# vgcreate --verbose --clustered n --metadatatype 2 --physicalextentsize 64M vg00 /dev/sda3
    Wiping cache of LVM-capable devices
    Wiping cache of LVM-capable devices
    Adding physical volume '/dev/sda3' to volume group 'vg00'
    Creating directory "/etc/lvm/archive"
    Archiving volume group "vg00" metadata (seqno 0).
    Creating directory "/etc/lvm/backup"
    Creating volume group backup "/etc/lvm/backup/vg00" (seqno 1).
  Volume group "vg00" successfully created
root@canopus:~#

Making /dev/sda2 available

/dev/sda2 is currently used as swap space. It's the same size as /dev/sda1. I want to deactivate the swap space here and make a copy of the root filesystem. First, though, there should be a different swap space make available.

Making our first logical volume: /dev/vg00/swap

Creating a logical volume is done with the lvcreate command. You want to decide a few things before creating one: how big should it be? What name should it have? Which volume group does it belong to?

Here's what I did to make /dev/vg00/swap:

root@canopus:~# lvcreate --test --verbose --size 512M --name swap --zero y vg00
  Test mode: Metadata will NOT be updated.
    Setting logging type to disk
    Finding volume group "vg00"
    Test mode: Skipping archiving of volume group.
    Creating logical volume swap
    Test mode: Skipping volume group backup.
    Found volume group "vg00"
  Aborting. Failed to activate new LV to wipe the start of it.
    Found volume group "vg00"
  Unable to deactivate failed new LV. Manual intervention required.
    Test mode: Wiping internal cache
    Wiping internal VG cache
root@canopus:~# lvcreate --verbose --size 512M --name swap --zero y vg00
    Setting logging type to disk
    Finding volume group "vg00"
    Archiving volume group "vg00" metadata (seqno 5).
    Creating logical volume swap
    Creating volume group backup "/etc/lvm/backup/vg00" (seqno 6).
    Found volume group "vg00"
    Creating vg00-swap
    Loading vg00-swap table (254:0)
    Resuming vg00-swap (254:0)
    Clearing start of logical volume "swap"
    Creating volume group backup "/etc/lvm/backup/vg00" (seqno 6).
  Logical volume "swap" created
root@canopus:~#

Reconfiguring swap location

After this logical volume has been created, we'll put the swap space signature on it using mkswap, activate it with swapon, deactivate the old one (swapoff), and update /etc/fstab to save the changes across a reboot:

root@canopus:~# mkswap /dev/vg00/swap
mkswap: /dev/vg00/swap: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.
Setting up swapspace version 1, size = 524284 KiB
no label, UUID=76b70850-c3ab-4512-b55b-0545f2690130
root@canopus:~# swapon /dev/vg00/swap
root@canopus:~# swapoff /dev/sda2
root@canopus:~# sed -e'/^\/dev\/sda2/s/^/# /' /etc/fstab > /tmp/fstab
root@canopus:~# echo '/dev/vg00/swap    none            swap    sw              0       0' >> /tmp/fstab
root@canopus:~# install -o root -g root -m 644 /tmp/fstab /etc/fstab
root@canopus:~# cat /etc/fstab
# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/root      /               ext2    noatime,errors=remount-ro 0 1
# /dev/sda2      none            swap    sw                0       0
tmpfs          /tmp            tmpfs   defaults          0       0
/dev/vg00/swap    none            swap    sw              0       0
root@canopus:~#

Just to be sure /dev/sda2 is not being used any more as swap space, have a look at /proc/swaps:

root@canopus:~# cat /proc/swaps 
Filename                                Type            Size    Used    Priority
/dev/dm-0                               partition       524280  0       -1
root@canopus:~#

/dev/sda2 is no longer being used for swap space and we can now put a backup root filesystem there.

Make a backup root filesystem

Now that we've stopped using /dev/sda2 for swap, let's put our backup root FS there. We'll format it with mkfs.ext2, mount it someplace convenient, and copy our files over.

create an ext2 filesystem on /dev/sda2

root@canopus:~# mkfs.ext2 -T ext2 /dev/sda2
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
32768 inodes, 131072 blocks
6553 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=134217728
4 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
       32768, 98304 

Writing inode tables: done
Writing superblocks and filesystem accounting information: done 

This filesystem will be automatically checked every 24 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
root@canopus:~#

populate the backup

I like to use /altroot for this purpose. My normal idiom for creating the directory underlying a mount point is to give it 000 (or d---------) permissions. Makes it obvious to me that something should be mounted there:

root@canopus:~# mkdir -m 0 /altroot
root@canopus:~# mount /dev/sda2 /altroot
root@canopus:~# cd /
root@canopus:/# find . -xdev -depth -print0 | cpio -pdmv0a /altroot
[...]
root@canopus:~# umount /altroot

With this done, you can use /dev/sda2 as your root filesystem by changing the "root=/dev/sda1" kernel command line prarameter (set in the bootloader) to "root=/dev/sda2".

Move stuff to LVM

Now for the good stuff. Our swap space is on an LVM logical volume. Next we'll move /home, /tmp, /usr, and /var to LVM.

create more logical volumes

We need a place to put our filesystems. We've already decided on sizes, so let's use lvcreate to make them:

root@canopus:/# lvcreate --test --verbose --size 64M --name home --zero y vg00
  Test mode: Metadata will NOT be updated.
    Setting logging type to disk
    Finding volume group "vg00"
    Test mode: Skipping archiving of volume group.
    Creating logical volume home
    Test mode: Skipping volume group backup.
    Found volume group "vg00"
  Aborting. Failed to activate new LV to wipe the start of it.
    Found volume group "vg00"
  Unable to deactivate failed new LV. Manual intervention required.
    Test mode: Wiping internal cache
    Wiping internal VG cache
root@canopus:/# lvcreate --verbose --size 64M --name home --zero y vg00
    Setting logging type to disk
    Finding volume group "vg00"
    Archiving volume group "vg00" metadata (seqno 8).
    Creating logical volume home
    Creating volume group backup "/etc/lvm/backup/vg00" (seqno 9).
    Found volume group "vg00"
    Creating vg00-home
    Loading vg00-home table (254:1)
    Resuming vg00-home (254:1)
    Clearing start of logical volume "home"
    Creating volume group backup "/etc/lvm/backup/vg00" (seqno 9).
  Logical volume "home" created
root@canopus:/#

Repeat this for /tmp (128Mbytes), /usr (192Mbytes), and /var (192Mbytes).

create filesystems to hold our files

We've got logical volumes now. Let's put filesystems on them so we can keep our files there.

root@canopus:/# for lvol in /dev/vg00/{home,tmp,usr,var}; do mkfs.ext2 -t ext3 -T default ${lvol}; done
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
4096 inodes, 16384 blocks
819 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=16777216
1 block group   
32768 blocks per group, 32768 fragments per group
4096 inodes per group

Writing inode tables: done
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 29 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux  
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
8192 inodes, 32768 blocks
1638 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=33554432
1 block group   
32768 blocks per group, 32768 fragments per group
8192 inodes per group

Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 29 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux  
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
12288 inodes, 49152 blocks
2457 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=50331648
2 block groups  
32768 blocks per group, 32768 fragments per group
6144 inodes per group
Superblock backups stored on blocks:
        32768   

Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 34 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux  
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
12288 inodes, 49152 blocks
2457 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=50331648
2 block groups  
32768 blocks per group, 32768 fragments per group
6144 inodes per group
Superblock backups stored on blocks:
        32768   

Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done 

This filesystem will be automatically checked every 21 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
root@canopus:/#

Copy existing files to new filesystems

We've created filesystems to hold our stuff. Now we need to put our stuff in there. Here's a little shell script procedure to do just that. This script's output is quite long, se I'll just skip showing it.

root@canopus:/# for fs in home tmp usr var; do 
                    mount -v /dev/vg00/${fs} /mnt; 
                    cd /${fs}; 
                    find . -xdev -depth -print0 | cpio -pdmv0a  /mnt; 
                    cd /; 
                    umount -v /mnt; 
                done

update /etc/fstab

Add the following lines to /etc/fstab. This ensures our new filesystems will be mounted and used the next time this machine reboots.

/dev/vg00/home  /home           ext3    defaults                        0       0
/dev/vg00/tmp   /tmp            ext3    defaults                        0       0
/dev/vg00/usr   /usr            ext3    defaults                        0       0
/dev/vg00/var   /var            ext3    defaults                        0       0

Also, comment out the line that lists /tmp on a tmpfs (RAM-backed) filesystem.