In this Juletip I will show you how you can combine multiple storage volumes into a single logical volume that you can format an mount in Linux.
Using this technique, I achieve a disk I/O throughput of more than 420-435 MB/s at the file system level using 4 disks compared to around 150-170 MB/s using a single disk.
In my examples, I will use Azure. However, this juletip works just as well on other cloud platforms such as AWS or GCP as well as on premises.
First, we need to attach the disks.
In Azure Portal, navigate to the VM and then click Disks
This will show the disks attached to your VM
In this case, I only have the OS disk attached.
We will now attach 5 new disks:
As we can see here, the single 400 GB disk comes with max IOPS of 2300 and max throughput of 150 MB/s.
So that is one way to add 400 GB storage.
We also add 4 disks of 100 GB each. These come with max IOPS of 500 and max throughput of 100 MB/s each.
While the aggregate IOPS of the 100 GB disks is slightly less than that of the single big disk (2000 vs 2300), the max throughput is significantly bigger (400 MB/s vs 150 MB/s).
This is exactly the tradeoff we are interested in when running a typical SAS workload because with SAS, raw throughput on sequential I/O is typically more important than IOPS.
Now let’s connect to the VM via SSH and login as root.
The command ‘lsblk’ lets us see attached disks:
~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 30G 0 disk ├─sda1 8:1 0 29.9G 0 part / ├─sda14 8:14 0 3M 0 part └─sda15 8:15 0 124M 0 part /boot/efi sdb 8:16 0 100G 0 disk sdc 8:32 0 100G 0 disk sdd 8:48 0 100G 0 disk sde 8:64 0 100G 0 disk
Notice the 4 disks of 100 GB.
These are the ones that we will combine into a single logical volume.
Before we can do that, we need to create a partition table on the disks.
We do that using the cfdisk command:
~# cfdisk /dev/sdb
We will just create a single partition utilizing all the space on the disk:
Disk: /dev/sdb Size: 100 GiB, 107374182400 bytes, 209715200 sectors Label: gpt, identifier: FBFC3E10-FC1A-8647-96BD-618BBC662E38 Device Start End Sectors Size Type >> /dev/sdb1 2048 209715166 209713119 100G Linux filesystem ┌─────────────────────────────────────────────────────────────────────────────┐ │Partition UUID: 9C66A040-5826-3147-BDDF-1456609BB07D │ │Partition type: Linux filesystem (0FC63DAF-8483-4772-8E79-3D69D8477DE4) │ └─────────────────────────────────────────────────────────────────────────────┘ [ Delete ] [ Resize ] [ Quit ] [ Type ] [ Help ] [ Write ] [ Dump ]
The lsblk command will show us the partition is made:
~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 30G 0 disk ├─sda1 8:1 0 29.9G 0 part / ├─sda14 8:14 0 3M 0 part └─sda15 8:15 0 124M 0 part /boot/efi sdb 8:16 0 100G 0 disk └─sdb1 8:17 0 100G 0 part sdc 8:32 0 100G 0 disk sdd 8:48 0 100G 0 disk sde 8:64 0 100G 0 disk
We will now use sfdisk to copy the partition table onto the remaining 3 disks:
~# for disk in /dev/sd[c-e]; do sfdisk -d /dev/sdb | sfdisk --force $disk; done Checking that no-one is using this disk right now ... OK Disk /dev/sdc: 100 GiB, 107374182400 bytes, 209715200 sectors Disk model: Virtual Disk Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes >>> Script header accepted. >>> Script header accepted. >>> Script header accepted. >>> Script header accepted. >>> Script header accepted. >>> Script header accepted. >>> Script header accepted. >>> Created a new GPT disklabel (GUID: FBFC3E10-FC1A-8647-96BD-618BBC662E38). /dev/sdc1: Created a new partition 1 of type 'Linux filesystem' and of size 100 GiB. /dev/sdc2: Done. New situation: Disklabel type: gpt Disk identifier: FBFC3E10-FC1A-8647-96BD-618BBC662E38 Device Start End Sectors Size Type /dev/sdc1 2048 209715166 209713119 100G Linux filesystem The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks. Checking that no-one is using this disk right now ... OK Disk /dev/sdd: 100 GiB, 107374182400 bytes, 209715200 sectors Disk model: Virtual Disk Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes >>> Script header accepted. >>> Script header accepted. >>> Script header accepted. >>> Script header accepted. >>> Script header accepted. >>> Script header accepted. >>> Script header accepted. >>> Created a new GPT disklabel (GUID: FBFC3E10-FC1A-8647-96BD-618BBC662E38). /dev/sdd1: Created a new partition 1 of type 'Linux filesystem' and of size 100 GiB. /dev/sdd2: Done. New situation: Disklabel type: gpt Disk identifier: FBFC3E10-FC1A-8647-96BD-618BBC662E38 Device Start End Sectors Size Type /dev/sdd1 2048 209715166 209713119 100G Linux filesystem The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks. Checking that no-one is using this disk right now ... OK Disk /dev/sde: 100 GiB, 107374182400 bytes, 209715200 sectors Disk model: Virtual Disk Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes >>> Script header accepted. >>> Script header accepted. >>> Script header accepted. >>> Script header accepted. >>> Script header accepted. >>> Script header accepted. >>> Script header accepted. >>> Created a new GPT disklabel (GUID: FBFC3E10-FC1A-8647-96BD-618BBC662E38). /dev/sde1: Created a new partition 1 of type 'Linux filesystem' and of size 100 GiB. /dev/sde2: Done. New situation: Disklabel type: gpt Disk identifier: FBFC3E10-FC1A-8647-96BD-618BBC662E38 Device Start End Sectors Size Type /dev/sde1 2048 209715166 209713119 100G Linux filesystem The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks.
We will verify using lsblk:
~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 30G 0 disk ├─sda1 8:1 0 29.9G 0 part / ├─sda14 8:14 0 3M 0 part └─sda15 8:15 0 124M 0 part /boot/efi sdb 8:16 0 100G 0 disk └─sdb1 8:17 0 100G 0 part sdc 8:32 0 100G 0 disk └─sdc1 8:33 0 100G 0 part sdd 8:48 0 100G 0 disk └─sdd1 8:49 0 100G 0 part sde 8:64 0 100G 0 disk └─sde1 8:65 0 100G 0 part
Now we need to make these partitions available for LVM. We do that by adding them as physical volumes:
~# pvcreate /dev/sd[b-e]1 Physical volume "/dev/sdb1" successfully created. Physical volume "/dev/sdc1" successfully created. Physical volume "/dev/sdd1" successfully created. Physical volume "/dev/sde1" successfully created.
We can list available physical volumes with the ‘pvs’ command:
~# pvs PV VG Fmt Attr PSize PFree /dev/sdb1 lvm2 --- <100.00g <100.00g /dev/sdc1 lvm2 --- <100.00g <100.00g /dev/sdd1 lvm2 --- <100.00g <100.00g /dev/sde1 lvm2 --- <100.00g <100.00g
Our logical volume must reside on a volume group. So we will now create a volume group containing the newly created physical volumes:
~# vgcreate vg1 /dev/sd[b-e]1 Volume group "vg1" successfully created
We can list volume groups using the ‘vgs’ command:
~# vgs VG #PV #LV #SN Attr VSize VFree vg1 4 0 0 wz--n- 399.98g 399.98g
We are now ready to create the actual volume group. We will create it using 4 stripes each with a stripe size of 256k:
~# lvcreate -i 4 -I 256k -n lv1 -L 399G vg1 Logical volume "lv1" created.
We can list logical volumes using the ‘lvs’ command:
~# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert lv1 vg1 -wi-a----- 399.00g
Before we can mount and use the logical volume we need to format it with a file system. I am using the XFS filesystem:
~# mkfs.xfs /dev/vg1/lv1 meta-data=/dev/vg1/lv1 isize=512 agcount=16, agsize=6537152 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=0 data = bsize=4096 blocks=104594432, imaxpct=25 = sunit=64 swidth=256 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=51071, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Discarding blocks...Done.
We are now ready to mount and use the logical volume:
~# mount /dev/vg1/lv1 /saswork/
Test write speed:
~# time head -c 200G /dev/zero | pv > /saswork/zero.dat 200GiB 0:07:50 [ 435MiB/s] [ <=> ] real 7m50.776s user 0m13.978s sys 3m38.363s
We just wrote 200 GB data with a throughput of 435 MB/s. Not bad.
Let’s see how fast we can read the file.
First we drop all cache to ensure that we actually read the file from the storage layer and not just from file system cache in RAM.
~# sync ; echo 3 > /proc/sys/vm/drop_caches
Then we measure read throughput:
~# time pv /saswork/zero.dat > /dev/null 200GiB 0:08:00 [ 426MiB/s] [=================================>] 100% real 8m0.308s user 0m0.861s sys 1m16.904s
We read the 200 GB file at 426 MB/s.
For comparison on a single storage volume, I was able to read and write at 153 MB/s and 167 MB/s respectively. (Data below)
The table below summarizes the results:
|
Throughput |
Read |
Write |
|
Single volume |
153 MB/s |
167 MB/s |
|
Logical volume striped over 4 volumes |
426 MB/s |
435 MB/s |
|
Speed increase |
178 % |
160 % |
Terminal log below:
(root@vm04) (2022-11-22 14:50:39) [0] ~# lvremove vg1 Do you really want to remove active logical volume vg1/lv1? [y/n]: y Logical volume "lv1" successfully removed (root@vm04) (2022-11-22 14:50:50) [0] ~# vgremove vg1 Volume group "vg1" successfully removed (root@vm04) (2022-11-22 14:50:55) [0] ~# pvremove /dev/sd[b-e]1 Labels on physical volume "/dev/sdb1" successfully wiped. Labels on physical volume "/dev/sdc1" successfully wiped. Labels on physical volume "/dev/sdd1" successfully wiped. Labels on physical volume "/dev/sde1" successfully wiped. (root@vm04) (2022-11-22 14:51:13) [0] ~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 30G 0 disk ├─sda1 8:1 0 29.9G 0 part / ├─sda14 8:14 0 3M 0 part └─sda15 8:15 0 124M 0 part /boot/efi sdb 8:16 0 100G 0 disk └─sdb1 8:17 0 100G 0 part sdc 8:32 0 100G 0 disk └─sdc1 8:33 0 100G 0 part sdd 8:48 0 100G 0 disk └─sdd1 8:49 0 100G 0 part sde 8:64 0 100G 0 disk └─sde1 8:65 0 100G 0 part (root@vm04) (2022-11-22 14:51:19) [0] ~# mkfs.xfs /dev/sdc1 meta-data=/dev/sdc1 isize=512 agcount=4, agsize=6553535 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=0 data = bsize=4096 blocks=26214139, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=12799, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Discarding blocks...Done. (root@vm04) (2022-11-22 14:51:31) [0] ~# mount /dev/sdc1 /saswork/ (root@vm04) (2022-11-22 14:51:38) [0] ~# time head -c 90G /dev/zero | pv > /saswork/zero.dat 90.0GiB 0:09:09 [ 167MiB/s] [ <=>] real 9m9.727s user 0m7.200s sys 1m53.040s (root@vm04) (2022-11-22 15:01:05) [0] ~# sync ; echo 3 > /proc/sys/vm/drop_caches (root@vm04) (2022-11-22 15:01:23) [0] ~# time pv /saswork/zero.dat > /dev/null 90.0GiB 0:10:01 [ 153MiB/s] [================================================================================>] 100% real 10m1.976s user 0m0.522s sys 0m38.955s
