Move /var to a new partition in VM

2021-Jan-19 • Tags: fedora

Well if there's not enough planning in building a resource it'll eat away our productive time when it fails or doesn't work as anticipated. In the current context the effort in migrating /var partition might be less but if not done properly may result in data loss!

Current setup before migrating /var to new partition:

1
-> cat /etc/os-release | grep PRETTY_NAME
2
PRETTY_NAME="Fedora 32 (Server Edition)"
3

4
-> lsblk
5
NAME               MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
6
sda                  8:0    0   80G  0 disk 
7
├─sda1               8:1    0    1G  0 part /boot
8
└─sda2               8:2    0   79G  0 part 
9
  ├─fedora-root    253:0    0   45G  0 lvm  /
10
  └─fedora-swap    253:1    0  7.9G  0 lvm  [SWAP]
11

12
-> df -h
13
Filesystem               Size  Used Avail Use% Mounted on
14
devtmpfs                 3.9G     0  3.9G   0% /dev
15
tmpfs                    3.9G     0  3.9G   0% /dev/shm
16
tmpfs                    3.9G  1.6M  3.9G   1% /run
17
tmpfs                    3.9G     0  3.9G   0% /sys/fs/cgroup
18
/dev/mapper/fedora-root   45G   16G   30G  36% /
19
tmpfs                    3.9G  4.0K  3.9G   1% /tmp
20
/dev/sda1               1014M  285M  730M  29% /boot
21
tmpfs                    786M     0  786M   0% /run/user/0

As you can infer from above there's no separate mount for /var and currently is part of /, however I intend to move /var to a different partition for future use. Being a VM running on KVM host below are the steps that'll be followed in brief:

• Create a qcow2 image and attach to the VM
• Create and mount xfs filesystem on newly added disk, rsync existing /var to newly mounted disc
• Add details in /etc/fstab to mount /var on reboot

1. Attach disk to VM §

On a sidenote, if performing partioning on an existing disk please follow this guide. Back to our scenario, create and attach a new disk to the guest machine and perform below on KVM machine

1
# Change below vars to your needs
2
-> disk_name=var-disk
3
-> disk_size=100G
4
-> vm_name=fedora-32
5
-> target_disk=sdb
6

7
# Get KVM default pool, if using some fancy directoy name please assign to 'pool_path' directly
8
-> pool_path=$(virsh pool-dumpxml default | grep -Po '(?<=path>)[[:alnum:]/.-]+(?=<)')
9

10
# Create 'qcow2' image with required size
11
-> qemu-img create -o preallocation=metadata -f qcow2 $pool_path/$disk_name $disk_size
12

13
# Attach newly created disk to VM
14
-> virsh attach-disk $vm_name --source $pool_path/$disk_name --target $target_disk --driver qemu --subdriver qcow2 --persistent

Verify the disk is created and attached to the guest (Fedora) machine

1
# On GUEST Machine verify disk is recognized
2
-> lsblk
3
NAME               MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
4
sda                  8:0    0   80G  0 disk 
5
├─sda1               8:1    0    1G  0 part /boot
6
└─sda2               8:2    0   79G  0 part 
7
  ├─fedora-root    253:0    0   45G  0 lvm  /
8
  └─fedora-swap    253:1    0  7.9G  0 lvm  [SWAP]
9
sdb                  8:16   0  100G  0 disk

2. Create a file system on the disk §

After adding disk to VM, you can follow this guide until creation of filesystem, cause in our scenrio we are creating an xfs filesystem. Perform mkfs.xfs /dev/vg1/lv1 on host VM and a new filesystem will be created on the disk.

If incase you want to use already existing disk and create xfs on it, wipe (WARNING: potential data loss) existing file system headers by running wipefs -a <disk> and create new filesystem on that.

1
# On Guest (VM) verify volume group is created
2
-> lsblk
3
NAME               MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
4
sda                  8:0    0   80G  0 disk 
5
├─sda1               8:1    0    1G  0 part /boot
6
└─sda2               8:2    0   79G  0 part 
7
  ├─fedora-root    253:0    0   45G  0 lvm  /
8
  └─fedora-swap    253:1    0  7.9G  0 lvm  [SWAP]
9
sdb                  8:16   0  100G  0 disk
10
└─vg1-lv1          253:2    0  100G  0 lvm

*If you wish not to go through all the hassle of creating LVM, you can simply run mkfs.xfs on new disk.

3. Sync /var to new disk §

We will mount new disk and to avoid any data writes we'll be dropping to runlevel 1 and perform rsync data of /var to mounted disk

1
-> mkdir /mnt/new-dir
2
-> mount /dev/mapper/vg1-lv1 /mnt/new-dir
3

4
# Drop to single user mode
5
-> telinit 1
6

7
# Check the runlevel and take note of last runlevel
8
-> who -r
9
         run-level 1  2021-01-20 07:49                   last=3
10

11
# rsync (or 'cp' also would work the same) 'var' contents to '/mnt/new-dir'
12
-> rsync -aqxp /var/* /mnt/new-dir/
13

14
# Rename '/var'
15
-> mv /var /var.old && mkdir /var
16

17
# Take note of UUID of LVM
18
-> UUID=$(blkid /dev/mapper/vg1-lv1 | grep -oP '(?<=UUID=").*?(?=")')
19

20
# Add entry in '/etc/fstab'
21
-> echo UUID=$UUID /var xfs defaults 0 0 >> /etc/fstab
22

23
# Unmount /mnt/new-dir and mount '/var' as mentioned in '/etc/fstab'
24
-> umount /mnt/new-dir
25
-> mount -a
26

27
# IMP: Restore SELinux labels on newly mounted partition
28
-> restorecon -R /var
29

30
# It's best to reboot the server to ascertain no errors
31
# Go to multi user mode (revert to last runlevel from line #9) and reboot the server
32
-> telinit 3 # ('last=3' in line #9, in GUI environment it's typically '5')
33
-> reboot

If everything checks out well, server should reboot without any issues with /var mounted as per /etc/fstab entries.

Verification and minimal troubleshoot §

After server restarts mount | grep var should be successfull, df -h should list /var being mounted on different filesystem and lsblk stating /var on lvm

Below is the info after migration:

1
-> mount | grep var
2
/dev/mapper/vg1-lv1 on /var type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
3

4
-> df -h
5
Filesystem               Size  Used Avail Use% Mounted on
6
devtmpfs                 3.9G     0  3.9G   0% /dev
7
tmpfs                    3.9G     0  3.9G   0% /dev/shm
8
tmpfs                    3.9G  1.6M  3.9G   1% /run
9
tmpfs                    3.9G     0  3.9G   0% /sys/fs/cgroup
10
/dev/mapper/fedora-root   45G   16G   30G  36% /
11
tmpfs                    3.9G  4.0K  3.9G   1% /tmp
12
/dev/sda1               1014M  285M  730M  29% /boot
13
tmpfs                    786M     0  786M   0% /run/user/0
14
/dev/mapper/vg1-lv1      100G  3.8G   97G   4% /var
15

16
-> lsblk
17
NAME               MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
18
sda                  8:0    0   80G  0 disk 
19
├─sda1               8:1    0    1G  0 part /boot
20
└─sda2               8:2    0   79G  0 part 
21
  ├─fedora-root    253:0    0   45G  0 lvm  /
22
  └─fedora-swap    253:1    0  7.9G  0 lvm  [SWAP]
23
sdb                  8:16   0  100G  0 disk 
24
└─vg1-lv1          253:2    0  100G  0 lvm  /var

If for some reason the server didn't come up as expected, login to VM via console connection (virt-manager) from KVM and perform below:

1
# Backup and remove newly added entry in /etc/fstab
2
-> cp /etc/fstab /etc/fstab.old
3
-> head -n -1 /etc/fstab.old > /etc/fstab
4

5
# Unmount '/var' and rename old directory (from previous step) to '/var'
6
-> umount /var && mv -f /var.old /var
7

8
# Reboot server and compare info against before migration, it should match
9
-> reboot

Many services use /var for log messages and storing data apart from configs, it's always a good idea particulary in server environments to have a healthy amount of free space available in /var directory.

All above steps are performed and verified on Fedora 32 Server Edition as part of making space for container images and related volume mounts.

Send an email for any comments. Kudos for making it to the end. Thanks!