Replacing physical disk in LVM

(note to self: add illustration to the whole process)

The situation

#

I recently move to Alma Linux to learn more about enterprise linux (RHEL like). During installation of Alma Linux, I noticed tha it created a logical volume (LV) by default for /home spanning across 2 disks, which made me nervous because I don’t know how to move /home to a separate disk like I used to do before with LVM.

The LV /dev/almalinux/home spans across 2 partitions on 2 physical disks: /dev/sdd - a 500GB SSD, and /dev/sdf - a 1.2TB SAS

Then I got more nervous when I got this in log: Device: /dev/sdf, SMART Failure: DATA CHANNEL IMPENDING FAILURE DATA ERROR RATE TOO HIGH. That’s why I have been avoiding using /home directory for any important task. Looks like it’s time time to learn to replace a disk that belongs to a LVM group.

Got stuck at the first step move the extents off of /dev/sdf1 - the failing disk: pmove /dev/sdf1 returns “No extents available for allocation.”

It’s because I can’t move 1TB space to 500GB eventhough there’s no file on that 1TB. Looks like I have to add a member to the group that’s big enough to store the extent. But how will I remove the member later?

Turned out it’s not like that. After going through RHEL fantastic documentation on LVM, looks like I just need to shrink the LV down before moving it to another disk.

Remove failing physical disk

#

Before doing this, backup all /home data, do all under root.

1. Show PV pvs -o+pv_used

PV         VG           Fmt  Attr PSize  PFree   Used
/dev/sdd1  almalinux    lvm2 a-   485G   0G      485G
/dev/sdf1  almalinux    lvm2 a-   1.1T   0G      1.1G

2. Unmount file system (FS) - in this case /home before we can remove the logical volume /dev/almalinux/home on /dev/sdf1. If the FS can’t be unmounted because it’s busy, has to switch to root sudo su; see what processes that’s using /home with lsof /home; then kill the processes with kill PID1 PID2 PID3.

Unmount /home. If it’s still busy, try lazy unmount umount -l /home so it’ll unmount whenever /home is available.

3. Shrink the LV almalinux/home so it’s small enough to move from /dev/sdf1 to /dev/sdd1 with lvreduce -L-1.1T /dev/almalinux/home. Note the -L flag indicates resizing by disk size, -l indicates resizing by logical extents. Now the LV only takes up a few GB on /dev/sdf1

4. Remove the LV from the group with lvremove /dev/almalinux/home

5. Now we’re ready to move the extents off of /dev/sdf1 with pvmove /dev/sdb1 pvs -o+pv_used shows

PV         VG           Fmt  Attr PSize  PFree   Used
/dev/sdd1  almalinux    lvm2 a-   485G   0G      485G
/dev/sdf1  almalinux    lvm2 a-   1.1T   1.1T      0G

6. Finally we can remove /dev/sdf1 from the almalinux group with vgreduce almalinux /dev/sdf1 and pvremove /dev/sdf1 pvs -o+pv_used shows

PV         VG           Fmt  Attr PSize  PFree   Used
/dev/sdd1  almalinux    lvm2 a-   485G   0G      485G

Add new physical disk

#

7. Add a new physical disk /dev/sdi1 to the group. The disk mustn’t have any partition table on it.

# initializes physical disk or partition as a LVM physical volume
pvcreate /dev/sdi1

8. grow the extent of the group to the new disk

lvextend -L+1G /dev/myvg/homevol

One side lesson

#

The management of logical and physical volumes above was certainly more complex and of a hassle than partitioning and formatting physical disks with fdisk and mkfs, as I am used to. This raises the question: why would we want to use LVM if it makes the administration process more complicated? However, after reading more into it, it does make sense why Enterprise Linux distros make it a standard. It all comes down to administrating large storage setups or frequently changing storage needs, which are more commonly found in business environments rather than a homelab.

1. With LVM we can easily resize logical volumes (like partitions) on the fly without needing to reboot the system. This is particularly useful if we need to dynamically allocate storage space based on changing workloads.
2. We can add more physical disks to an existing volume group to expand the total storage pool. This allows a seamless increase of storage capacity without needing to repartition individual disks or recreate file systems.
3. The layer of abstraction LVM provides simplifies storage management and features like mirroring and striping can enhance data redundancy and storage performance.

This reminds me that as a student sysadmin, it’s important to keep in mind that many of the methodologies, practices, and solutions I’m learning are designed for high-workload environments that require high availability and continuity. While it may be tempting to opt for easy or convenient solutions, it’s important to consider the long-term implications and potential trade-offs. By learning to work within these constraints, I’ll be better prepared to handle the challenges of managing large-scale storage setups and ensuring that critical systems remain up and running 24/7.