Published on

Setup a 3 Node Replicated Storage Volume with GlusterFS

Authors

In one of my earlier posts on GlusterFS, we went through the steps on how to setup a Distributed Storage Volume, where the end result was to have scalable storage, where size was the requirement.

ruanbekker-cheatsheets

What will we be doing today with GlusterFS?

Today, we will be going through the steps on how to setup a Replicated Storage Volume with GlusterFS, where we will have 3 GlusterFS Nodes, and using the replication factor of 3.

Replication Factor of 3:

In other words, having 3 copies of our data and in our case, since we will have 3 nodes in our cluster, a copy of our data will reside on each node.

What about Split-Brain:

In Clustering, we get the term Split-Brain, where a node dies or leaves the cluster, the cluster reforms itself with the available nodes and then during this reformation, instead of the remaining nodes staying with the same cluster, 2 subset of cluster are created, and they are not aware of each other, which causes data corruption, here's a great resource on Split-Brain

To prevent Split-Brain in GlusterFS, we can setup a Arbiter Volume. In a Replica Count of 3 and Arbiter count of 1: 2 Nodes will hold the replicated data, and the 1 Node which will be the Arbiter node, will only host the file/directory names and metadata but not any data. I will write up an article on this in the future.

Getting Started:

Let's get started on setting up a 3 Node Replicated GlusterFS. Each node will have an additional drive that is 50GB in size, which will be part of our GlusterFS Replicated Volume. I will also be using Ubuntu 16.04 as my linux distro.

Preparing DNS Resolution:

I will install GlusterFS on each node, and in my setup I have the following DNS entries:

  • gfs01 (10.0.0.2)
  • gfs02 (10.0.0.3)
  • gfs03 (10.0.0.4)

Preparing our Secondary Drives:

I will be formatting my drives with XFS. Listing our block volumes:

$ lsblk
NAME MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vdb  253:16   0 46.6G  0 disk
vda  253:0    0 18.6G  0 disk /

Creating the FileSystem with XFS, which we will be running on each node:

$ mkfs.xfs /dev/vdb

Then creating the directories where our bricks will reside, and also add an entry to our /etc/fstab so that our disk gets mounted when the operating system boots:

# node: gfs01
$ mkdir /gluster/bricks/1 -p
$ echo '/dev/vdb /gluster/bricks/1 xfs defaults 0 0' >> /etc/fstab
$ mount -a
$ mkdir /gluster/bricks/1/brick

# node: gfs02
$ mkdir /gluster/bricks/2 -p
$ echo '/dev/vdb /gluster/bricks/2 xfs defaults 0 0' >> /etc/fstab
$ mount -a
$ mkdir /gluster/bricks/2/brick

# node: gfs03
$ mkdir /gluster/bricks/3 -p
$ echo '/dev/vdb /gluster/bricks/3 xfs defaults 0 0' >> /etc/fstab
$ mount -a
$ mkdir /gluster/bricks/3/brick

After this has been done, we should see that the disks are mounted, for example on node: gfs01:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda         18G  909M   17G   3% /
/dev/vdb         47G   80M   47G   1% /gluster/bricks/1

Installing GlusterFS on Each Node:

Installing GlusterFS, repeat this on all 3 Nodes:

$ apt update && sudo apt upgrade -y
$ apt install xfsprogs attr glusterfs-server glusterfs-common glusterfs-client -y
$ systemctl enable glusterfs-server

In order to add the nodes to the trusted storage pool, we will have to add them by using gluster peer probe. Make sure that you can resolve the hostnames to the designated IP Addresses, and that traffic is allowed.

$ gluster peer probe gfs01
$ gluster peer probe gfs02
$ gluster peer probe gfs03

Now that we have added our nodes to our trusted storage pool, lets verify that by listing our pool:

$ gluster pool list
UUID                                    Hostname                State
f63d0e77-9602-4024-8945-5a7f7332bf89    gfs02                   Connected
2d4ac6c1-0611-4e2e-b4af-9e4aa8c1556d    gfs03                   Connected
6a604cd9-9a9c-406d-b1b7-69caf166a20e    localhost               Connected

Great! All looks good.

Create the Replicated GlusterFS Volume:

Let's create our Replicated GlusterFS Volume, named gfs:

$ gluster volume create gfs \
  replica 3 \
  gfs01:/gluster/bricks/1/brick \
  gfs02:/gluster/bricks/2/brick \
  gfs03:/gluster/bricks/2/brick 

volume create: gfs: success: please start the volume to access data

Now that our volume is created, lets list it to verify that it is created:

$ gluster volume list
gfs

Now, start the volume:

$ gluster volume start gfs
volume start: gfs: success

View the status of our volume:

$ gluster volume status gfs
Status of volume: gfs
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick gfs01:/gluster/bricks/1/brick         49152     0          Y       6450
Brick gfs02:/gluster/bricks/2/brick         49152     0          Y       3460
Brick gfs03:/gluster/bricks/3/brick         49152     0          Y       3309

Next, view the volume inforation:

$ gluster volume info gfs

Volume Name: gfs
Type: Replicate
Volume ID: 6f827df4-6df5-4c25-99ee-8d1a055d30f0
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: gfs01:/gluster/bricks/1/brick
Brick2: gfs02:/gluster/bricks/2/brick
Brick3: gfs03:/gluster/bricks/3/brick

Security:

From a GlusterFS level, it will allow clients to connect by default. To authorize these 3 nodes to connect to the GlusterFS Volume:

$ gluster volume set gfs auth.allow 10.0.0.2,10.0.0.3,10.0.0.4

Then if you would like to remove this rule:

$ gluster volume set gfs auth.allow *

Mount the GlusterFS Volume to the Host:

Mount the GlusterFS Volume to each node, so we will have to mount it to each node, and also append it to our /etc/fstab file so that it mounts on boot:

$ echo 'localhost:/gfs /mnt glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab
$ mount.glusterfs localhost:/gfs /mnt

Verify the Mounted Volume:

Check the mounted disks, and you will find that the Replicated GlusterFS Volume is mounted on our /mnt partition.

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda         18G  909M   17G   3% /
/dev/vdb         47G   80M   47G   1% /gluster/bricks/1
localhost:/gfs   47G   80M   47G   1% /mnt

You will note that GlusterFS Volume has a total size of 47GB usable space, which is the same size as one of our disks, but that is because we have a replicated volume with a replication factor of 3: (47 * 3 / 3)

Now we have a Storage Volume which has 3 Replicas, one copy on each node, which allows us Data Durability on our Storage.

Thank You

Thanks for reading, feel free to check out my website, and subscribe to my newsletter or follow me at @ruanbekker on Twitter.

Buy Me A Coffee
Discuss on Twitter • View on GitHub