- Published on
Setting up a Docker Swarm Cluster on 3 RaspberryPi Nodes
- Authors
- Name
- Ruan Bekker
- @ruanbekker
As the curious person that I am, I like to play around with new stuff that I stumble upon, and one of them was having a docker swarm cluster running on 3 Raspberry Pi's on my LAN.
The idea is to have 3 Raspberry Pi's (Model 3 B), a Manager Node, and 2 Worker Nodes, each with a 32 GB SanDisk SD Card, which I will also be part of a 3x Replicated GlusterFS Volume that will come in handy later for some data that needs persistent data.
More Inforamtion on: Docker Swarm
Provision Raspbian on each RaspberryPi
Grab the Latest Raspbian Lite ISO and the following source will help provisioning your RaspberryPi with Raspbian.
Installing Docker on Raspberry PI
On each node, run the following to install docker, and also add your user to the docker group, so that you can run docker commands with a normal user:
$ apt-get update && sudo apt-get upgrade -y
$ sudo apt-get remove docker.io
$ curl https://get.docker.com | sudo bash
$ sudo usermod -aG docker pi
If you have an internal DNS Server, set an A Record for each node, or for simplicity, set your hosts file on each node so that your hostname for each node responds to it's provisioned IP Address:
$ cat /etc/hosts
192.168.0.2 rpi-01
192.168.0.3 rpi-02
192.168.0.4 rpi-03
Also, to have passwordless SSH, from each node:
$ ssh-keygen -t rsa
$ ssh-copy-id rpi-01
$ ssh-copy-id rpi-02
$ ssh-copy-id rpi-03
Initialize the Swarm
Time to set up our swarm. As we have more than one network interface, we will need to setup our swarm by specifying the IP Address of our network interface that is accessible from our LAN:
$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr a1:12:bc:d3:cd:4d
inet addr:192.168.0.2 Bcast:192.168.0.255 Mask:255.255.255.0
Now that we have our IP Address, initialize the swarm on the manager node:
pi@rpi-01:~ $ docker swarm init --advertise-addr 192.168.0.2
Swarm initialized: current node (siqyf3yricsvjkzvej00a9b8h) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join \
--token SWMTKN-1-0eith07xkcg93lzftuhjmxaxwfa6mbkjsmjzb3d3sx9cobc2zp-97s6xzdt27y2gk3kpm0cgo6y2 \
192.168.0.2:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
Then from rpi-02 join the manager node of the swarm:
pi@rpi-02:~ $ docker swarm join --token SWMTKN-1-0eith07xkcg93lzftuhjmxaxwfa6mbkjsmjzb3d3sx9cobc2zp-97s6xzdt27y2gk3kpm0cgo6y2 192.168.0.2:2377
This node joined a swarm as a worker.
Then from rpi-03 join the manager node of the swarm:
pi@rpi-03:~ $ docker swarm join --token SWMTKN-1-0eith07xkcg93lzftuhjmxaxwfa6mbkjsmjzb3d3sx9cobc2zp-97s6xzdt27y2gk3kpm0cgo6y2 192.168.0.2:2377
This node joined a swarm as a worker.
Then from the manager node: rpi-01, ensure that the nodes are checked in:
pi@rpi-01:~ $ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
62s7gx1xdm2e3gp5qoca2ru0d rpi-03 Ready Active
6fhyfy9yt761ar9pl84dkxck3 * rpi-01 Ready Active Leader
pg0nyy9l27mtfc13qnv9kywe7 rpi-02 Ready Active
Setting Up a Replicated GlusterFS Volume
I have decided to setup a replicated glusterfs volume to have data replicated throughout the cluster if I would like to have some persistent data. From each node, install the GlusterFS Client and Server:
$ sudo apt install glusterfs-server glusterfs-client -y && sudo systemctl enable glusterfs-server
Probe the other nodes from the manager node:
pi@rpi-01:~ $ sudo gluster peer probe rpi-02
peer probe: success.
pi@rpi-01:~ $ sudo gluster peer probe rpi-03
peer probe: success.
Ensure that we can see all 3 nodes in our GlusterFS Pool:
pi@rpi-01:~ $ sudo gluster pool list
UUID Hostname State
778c7463-ba48-43de-9f97-83a960bba99e rpi-02 Connected
00a20a3c-5902-477e-a8fe-da35aa955b5e rpi-03 Connected
d82fb688-c50b-405d-a26f-9cb2922cce75 localhost Connected
From each node, create the directory where GlusterFS will store the data for the bricks that we will specify when creating the volume:
pi@rpi-01:~ $ sudo mkdir -p /gluster/brick
pi@rpi-02:~ $ sudo mkdir -p /gluster/brick
pi@rpi-03:~ $ sudo mkdir -p /gluster/brick
Next, create a 3 Way Replicated GlusterFS Volume:
pi@rpi-01:~ $ sudo gluster volume create rpi-gfs replica 3 \
rpi-01:/gluster/brick \
rpi-02:/gluster/brick \
rpi-03:/gluster/brick \
force
volume create: rpi-gfs: success: please start the volume to access data
Start the GlusterFS Volume:
pi@rpi-01:~ $ sudo gluster volume start rpi-gfs
volume start: rpi-gfs: success
Verify the GlusterFS Volume Info, and from the below output you will see that the volume is replicated 3 ways from the 3 bricks that we specified
pi@rpi-01:~ $ sudo gluster volume info
Volume Name: rpi-gfs
Type: Replicate
Volume ID: b879db15-63e9-44ca-ad76-eeaa3e247623
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: rpi-01:/gluster/brick
Brick2: rpi-02:/gluster/brick
Brick3: rpi-03:/gluster/brick
Mount the GlusterFS Volume on each Node, first on rpi-01:
pi@rpi-01:~ $ sudo umount /mnt
pi@rpi-01:~ $ sudo echo 'localhost:/rpi-gfs /mnt glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab
pi@rpi-01:~ $ sudo mount.glusterfs localhost:/rpi-gfs /mnt
pi@rpi-01:~ $ sudo chown -R pi:docker /mnt
Then on rpi-02:
pi@rpi-02:~ $ sudo umount /mnt
pi@rpi-02:~ $ sudo echo 'localhost:/rpi-gfs /mnt glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab
pi@rpi-02:~ $ sudo mount.glusterfs localhost:/rpi-gfs /mnt
pi@rpi-02:~ $ sudo chown -R pi:docker /mnt
And lastly on rpi-03:
pi@rpi-03:~ $ sudo umount /mnt
pi@rpi-03:~ $ sudo echo 'localhost:/rpi-gfs /mnt glusterfs defaults,_netdev,backupvolfile-server=localhost 0 0' >> /etc/fstab
pi@rpi-03:~ $ sudo mount.glusterfs localhost:/rpi-gfs /mnt
pi@rpi-03:~ $ sudo chown -R pi:docker /mnt
Then your GlusterFS Volume will be mounted on all the nodes, and when a file is written to the /mnt/ partition, data will be replicated to all the nodes in the Cluster:
pi@rpi-01:~ $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 30G 4.5G 24G 16% /
localhost:/rpi-gfs 30G 4.5G 24G 16% /mnt
Create a Web Service on Docker Swarm:
Let's create a Web Service in our Swarm, called web and by specifying 1 replica and publishing the exposed port 80 to our containers port 80:
pi@rpi-01:~ $ docker service create --name web --replicas 1 --publish 80:80 hypriot/rpi-busybox-httpd
vsvyanuw6q6yf4jr52m5z7vr1
Verifying that our Service is Started and equals to the desired replica count:
pi@rpi-01:~ $ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
vsvyanuw6q6y web replicated 1/1 hypriot/rpi-busybox-httpd:latest *:891->80/tcp
Inspecting the Service:
pi@rpi-01:~ $ docker service inspect web
[
{
"ID": "vsvyanuw6q6yf4jr52m5z7vr1",
"Version": {
"Index": 2493
},
"CreatedAt": "2017-07-16T21:20:00.017836646Z",
"UpdatedAt": "2017-07-16T21:20:00.026359794Z",
"Spec": {
"Name": "web",
"Labels": {},
"TaskTemplate": {
"ContainerSpec": {
"Image": "hypriot/rpi-busybox-httpd:latest@sha256:c00342f952d97628bf5dda457d3b409c37df687c859df82b9424f61264f54cd1",
"StopGracePeriod": 10000000000,
"DNSConfig": {}
},
"Resources": {
"Limits": {},
"Reservations": {}
},
"RestartPolicy": {
"Condition": "any",
"Delay": 5000000000,
"MaxAttempts": 0
},
"Placement": {},
"ForceUpdate": 0
},
"Mode": {
"Replicated": {
"Replicas": 1
}
},
"UpdateConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"RollbackConfig": {
"Parallelism": 1,
"FailureAction": "pause",
"Monitor": 5000000000,
"MaxFailureRatio": 0,
"Order": "stop-first"
},
"EndpointSpec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 80,
"PublishedPort": 80,
"PublishMode": "ingress"
}
]
}
},
"Endpoint": {
"Spec": {
"Mode": "vip",
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 80,
"PublishedPort": 80,
"PublishMode": "ingress"
}
]
},
"Ports": [
{
"Protocol": "tcp",
"TargetPort": 80,
"PublishedPort": 80,
"PublishMode": "ingress"
}
],
"VirtualIPs": [
{
"NetworkID": "zjerz0xsw39icnh24enja4cgk",
"Addr": "10.255.0.13/16"
}
]
}
}
]
Docker Swarm's Routing mesh takes care of the internal routing, so requests will respond even if the container is not running on the node that you are making the request against.
With that said, verifying on which node our service is running:
pi@rpi-01:~ $ docker service ps web
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
sd67cd18s5m0 web.1 hypriot/rpi-busybox-httpd:latest rpi-02 Running Running 2 minutes ago
When we make a HTTP Request to one of these Nodes IP Addresses, our request will be responded with this awesome static page:
We can see we only have one container in our swarm, let's scale that up to 3 containers:
pi@rpi-01:~ $ docker service scale web01=3
web01 scaled to 3
Now that the service is scaled to 3 containers, requests will be handled using the round-robin algorithm. To ensured that the service scaled, we will see that we will have 3 replicas:
pi@rpi-01:~ $ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
vsvyanuw6q6y web replicated 3/3 hypriot/rpi-busybox-httpd:latest *:891->80/tcp
Verifying where these containers are running on:
pi@rpi-01:~ $ docker service ps web01
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
sd67cd18s5m0 web.1 hypriot/rpi-busybox-httpd:latest rpi-02 Running Running 2 minutes ago
ope3ya7hh9j4 web.2 hypriot/rpi-busybox-httpd:latest rpi-03 Running Running 30 seconds ago
07m1ww7ptxro web.3 hypriot/rpi-busybox-httpd:latest rpi-01 Running Running 28 seconds ago
Lastly, removing the service from our swarm:
pi@rpi-01:~ $ docker service rm web01
web01
Massive Thanks:
a Massive thanks to Alex Ellis for mentioning me on one of his blogposts:
Thank You
Thanks for reading, if you like my content, feel free to check out my website, and subscribe to my newsletter or follow me at @ruanbekker on Twitter.
- Linktree: https://go.ruan.dev/links
- Patreon: https://go.ruan.dev/patreon
