- Published on
Create a RAID5 Array with mdadm on Linux
- Authors
- Name
- Ruan Bekker
- @ruanbekker
In this tutorial we will setup a RAID5 array, which is striping across multiple drives with distributed paritiy, which is good for redundancy. We will be using Ubuntu for our Linux Distribution, but the technique applies to other Linux Distributions as well.
What are we trying to achieve
We will run a server with one root disk and 6 extra disks, where we will first create our raid5 array with three disks, then I will show you how to expand your raid5 array by adding three other disks.
Things fail all the time, and it's not fun when hard drives breaks, therefore we want to do our best to prevent our applications from going down due to hardware failures. To achieve data redundancy, we want to use three hard drives, which we want to add into a raid configuration that will proviide us:
- striping, which is the technique of segmenting logically sequential data, so that consecutive segments are stored on different physical storage devices.
- distributed parity, where parity data are distributed between the physical disks, where there is only one parity block per disk, this provide protection against one physical disk failure, where the minimum number of disks are three.
This is how a RAID5 array looks like (image from diskpart.com):
Hardware Overview
We will have a Linux server with one root disk and six extra disks:
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
xvdb 202:16 0 10G 0 disk
xvdc 202:32 0 10G 0 disk
xvdd 202:48 0 10G 0 disk
xvde 202:64 0 10G 0 disk
xvdf 202:80 0 10G 0 disk
xvdg 202:96 0 10G 0 disk
Dependencies
We require mdadm to create our raid configuration:
$ sudo apt update
$ sudo apt install mdadm -y
Format Disks
First we will format and partition the following disks: /dev/xvdb, /dev/xvdc, /dev/xvdd, I will demonstrate the process for one disk, but repeat them for the other as well:
$ fdisk /dev/xvdc
Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
The old ext4 signature will be removed by a write command.
Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x26a2d2f6.
Command (m for help): n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-20971519, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-20971519, default 20971519):
Created a new partition 1 of type 'Linux' and of size 10 GiB.
Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): fd
Changed type of partition 'Linux' to 'Linux raid autodetect'.
Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
Create RAID5 Array
Using mdadm, create the /dev/md0 device, by specifying the raid level and the disks that we want to add to the array:
$ mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/xvdb1 /dev/xvdc1 /dev/xvdd1
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
Now that our device has been added, we can monitor the process:
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 xvdd1[3] xvdc1[1] xvdb1[0]
20951040 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [UU_]
[==>..................] recovery = 11.5% (1212732/10475520) finish=4.7min speed=32103K/sec
unused devices: <none>
As you can see, currently its at 11.5%, give it some time to let it complete, you should treat the following as a completed state:
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 xvdd1[3] xvdc1[1] xvdb1[0]
20951040 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
unused devices: <none>
We can also inspect devices with mdadm:
$ mdadm -E /dev/xvd[b-d]1
/dev/xvdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : ea997bce:a530519c:ae41022e:0f4306bf
Name : ip-172-31-3-57:0 (local to host ip-172-31-3-57)
Creation Time : Wed Jan 12 13:36:39 2022
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 20951040 (9.99 GiB 10.73 GB)
Array Size : 20951040 (19.98 GiB 21.45 GB)
Data Offset : 18432 sectors
Super Offset : 8 sectors
Unused Space : before=18280 sectors, after=0 sectors
State : clean
Device UUID : 8305a179:3ef96520:6c7b41dd:bdc7401f
Update Time : Wed Jan 12 13:42:14 2022
Bad Block Log : 512 entries available at offset 136 sectors
Checksum : 1f9b4887 - correct
Events : 18
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
To get information about your raid5 device:
$ mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Wed Jan 12 13:36:39 2022
Raid Level : raid5
Array Size : 20951040 (19.98 GiB 21.45 GB)
Used Dev Size : 10475520 (9.99 GiB 10.73 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Wed Jan 12 13:42:14 2022
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : resync
Name : ip-172-31-3-57:0 (local to host ip-172-31-3-57)
UUID : ea997bce:a530519c:ae41022e:0f4306bf
Events : 18
Number Major Minor RaidDevice State
0 202 17 0 active sync /dev/xvdb1
1 202 33 1 active sync /dev/xvdc1
3 202 49 2 active sync /dev/xvdd1
Create Filesystems
We will use our /dev/md0 device and create a ext4 filesystem:
$ mkfs.ext4 /dev/md0
mke2fs 1.45.5 (07-Jan-2020)
Creating filesystem with 5237760 4k blocks and 1310720 inodes
Filesystem UUID: 579f045e-d270-4ff2-b36b-8dc506c27c5f
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
We can then verify that by looking at our block devices using lsblk:
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
xvdb 202:16 0 10G 0 disk
└─xvdb1 202:17 0 10G 0 part
└─md0 9:0 0 20G 0 raid5
xvdc 202:32 0 10G 0 disk
└─xvdc1 202:33 0 10G 0 part
└─md0 9:0 0 20G 0 raid5
xvdd 202:48 0 10G 0 disk
└─xvdd1 202:49 0 10G 0 part
└─md0 9:0 0 20G 0 raid5
xvde 202:64 0 10G 0 disk
xvdf 202:80 0 10G 0 disk
xvdg 202:96 0 10G 0 disk
Now we can mount our device to /mnt:
$ mount /dev/md0 /mnt
We can verify that the device is mounted by using df:
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 7.7G 1.5G 6.3G 19% /
/dev/md0 20G 45M 19G 1% /mnt
To persist the device across reboots, add it to the /etc/fstab file:
$ cat /etc/fstab
/dev/md0 /mnt ext4 defaults 0 0
Now our filesystem which is mounted at /mnt is ready to be used.
RAID Configuration (across reboots)
By default RAID doesn’t have a config file, therefore we need to save it manually. If this step is not followed RAID device will not be in md0, but perhaps something else.
So, we must have to save the configuration to persist across reboots, when it reboot it gets loaded to the kernel and RAID will also get loaded.
$ mdadm --detail --scan --verbose >> /etc/mdadm.conf
Note: Saving the configuration will keep the RAID level stable in the md0 device.
Adding Spare Devices
Earlier I mentioned that we have spare disks that we can use to expand our raid device. After they have been formatted we can add them as spare devices to our raid setup:
$ mdadm --add /dev/md0 /dev/xvde1 /dev/xvdf1 /dev/xvdg1
mdadm: added /dev/xvde1
mdadm: added /dev/xvdf1
mdadm: added /dev/xvdg1
Verify our change by viewing the detail of our device:
$ mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Wed Jan 12 13:36:39 2022
Raid Level : raid5
Array Size : 20951040 (19.98 GiB 21.45 GB)
Used Dev Size : 10475520 (9.99 GiB 10.73 GB)
Raid Devices : 3
Total Devices : 6
Persistence : Superblock is persistent
Update Time : Wed Jan 12 14:28:23 2022
State : clean
Active Devices : 3
Working Devices : 6
Failed Devices : 0
Spare Devices : 3
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : resync
Name : ip-172-31-3-57:0 (local to host ip-172-31-3-57)
UUID : ea997bce:a530519c:ae41022e:0f4306bf
Events : 27
Number Major Minor RaidDevice State
0 202 17 0 active sync /dev/xvdb1
1 202 33 1 active sync /dev/xvdc1
3 202 49 2 active sync /dev/xvdd1
4 202 65 - spare /dev/xvde1
5 202 81 - spare /dev/xvdf1
6 202 97 - spare /dev/xvdg1
As you can see it's only spares at this moment, we can use the spares for data storage, by growing our device:
$ mdadm --grow --raid-devices=6 /dev/md0
Verify:
$ mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Wed Jan 12 13:36:39 2022
Raid Level : raid5
Array Size : 20951040 (19.98 GiB 21.45 GB)
Used Dev Size : 10475520 (9.99 GiB 10.73 GB)
Raid Devices : 6
Total Devices : 6
Persistence : Superblock is persistent
Update Time : Wed Jan 12 15:15:31 2022
State : clean, reshaping
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : resync
Reshape Status : 0% complete
Delta Devices : 3, (3->6)
Name : ip-172-31-3-57:0 (local to host ip-172-31-3-57)
UUID : ea997bce:a530519c:ae41022e:0f4306bf
Events : 36
Number Major Minor RaidDevice State
0 202 17 0 active sync /dev/xvdb1
1 202 33 1 active sync /dev/xvdc1
3 202 49 2 active sync /dev/xvdd1
6 202 97 3 active sync /dev/xvdg1
5 202 81 4 active sync /dev/xvdf1
4 202 65 5 active sync /dev/xvde1
Wait for the raid to rebuild, by viewing the mdstat::
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 xvdg1[6] xvdf1[5] xvde1[4] xvdd1[3] xvdc1[1] xvdb1[0]
20951040 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6] [UUUUUU]
[>....................] reshape = 0.7% (76772/10475520) finish=18.0min speed=9596K/sec
unused devices: <none>
Resizing our Filesystem
Once we added the spares and growed our device, we need to run integrity checks, then we can resize the volume. But first, we need to unmount our filesystem:
$ umount /mnt
Run a integrity check:
$ e2fsck -f /dev/md0
e2fsck 1.45.5 (07-Jan-2020)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/md0: 12/1310720 files (0.0% non-contiguous), 126323/5237760 blocks
Once that has passed, resize the file system:
$ resize2fs /dev/md0
resize2fs 1.45.5 (07-Jan-2020)
Resizing the filesystem on /dev/md0 to 13094400 (4k) blocks.
The filesystem on /dev/md0 is now 13094400 (4k) blocks long.
Then we remount our filesystem:
$ mount /dev/md0 /mnt
After the filesystem has been mounted, we can view the disk size and confirm that the size increased:
$ df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/md0 50G 52M 47G 1% /mnt
Thank You
Thanks for reading, if you like my content, feel free to check out my website, and subscribe to my newsletter or follow me at @ruanbekker on Twitter.
- Linktree: https://go.ruan.dev/links
- Patreon: https://go.ruan.dev/patreon
