Published on

Setup a Concourse-CI Server on Ubuntu 16

Authors

Concourse is a Pipeline Based Continious Integration system written in Go

Resources:

A Newer Version

A newer up to date version for concourse v6 is available:

What is Concourse CI:

Concourse CI is a Continious Integration Platform. Concourse enables you to construct pipelines with a yaml configuration that can consist out of 3 core concepts, tasks, resources, and jobs that compose them. For more information about this have a look at their docs

What will we be doing today

We will setup a Concourse Server on Ubuntu 16.04 and run the traditional Hello, World pipeline

Setup the Server:

Concourse needs PostgresSQL 9.3+

apt update && apt upgrade -y
apt install postgresql postgresql-contrib -y
systemctl enable postgresql

Create the Database and User for Concourse on Postgres:

sudo -u postgres createuser concourse
sudo -u postgres createdb --owner=concourse atc

Download the Concourse and Fly Cli Binaries:

wget https://github.com/concourse/concourse/releases/download/v4.2.2/concourse_linux_amd64
wget https://github.com/concourse/concourse/releases/download/v4.2.2/fly_linux_amd64
chmod +x concourse_linux_amd64 fly_linux_amd64
mv concourse_linux_amd64 /usr/bin/concourse
mv fly_linux_amd64 /usr/bin/fly

Create the Encryption Keys:

mkdir /etc/concourse
ssh-keygen -t rsa -q -N '' -f /etc/concourse/tsa_host_key
ssh-keygen -t rsa -q -N '' -f /etc/concourse/worker_key
ssh-keygen -t rsa -q -N '' -f /etc/concourse/session_signing_key
cp /etc/concourse/worker_key.pub /etc/concourse/authorized_worker_keys

Concourse Web Process Configuration:

/etc/concourse/web_environment
CONCOURSE_ADD_LOCAL_USER=ruan:pass
CONCOURSE_SESSION_SIGNING_KEY=/etc/concourse/session_signing_key
CONCOURSE_TSA_HOST_KEY=/etc/concourse/tsa_host_key
CONCOURSE_TSA_AUTHORIZED_KEYS=/etc/concourse/authorized_worker_keys
CONCOURSE_POSTGRES_HOST=127.0.0.1
CONCOURSE_POSTGRES_USER=concourse
CONCOURSE_POSTGRES_PASSWORD=concourse
CONCOURSE_POSTGRES_DATABASE=atc
CONCOURSE_MAIN_TEAM_LOCAL_USER=ruan
CONCOURSE_EXTERNAL_URL=http://10.20.30.40:8080

Concourse Worker Process Configuration:

/etc/concourse/worker_environment
CONCOURSE_WORK_DIR=/var/lib/concourse
CONCOURSE_TSA_HOST=127.0.0.1:2222
CONCOURSE_TSA_PUBLIC_KEY=/etc/concourse/tsa_host_key.pub
CONCOURSE_TSA_WORKER_PRIVATE_KEY=/etc/concourse/worker_key

Create a Concourse user:

mkdir /var/lib/concourse
sudo adduser --system --group concourse
sudo chown -R concourse:concourse /etc/concourse /var/lib/concourse
sudo chmod 600 /etc/concourse/*_environment

Create SystemD Unit Files, first for the Web Service:

/etc/systemd/system/concourse-web.service
[Unit]
Description=Concourse CI web process (ATC and TSA)
After=postgresql.service

[Service]
User=concourse
Restart=on-failure
EnvironmentFile=/etc/concourse/web_environment
ExecStart=/usr/bin/concourse web

[Install]
WantedBy=multi-user.target

Then the SystemD Unit File for the Worker Service:

/etc/systemd/system/concourse-worker.service
[Unit]
Description=Concourse CI worker process
After=concourse-web.service

[Service]
User=root
Restart=on-failure
EnvironmentFile=/etc/concourse/worker_environment
ExecStart=/usr/bin/concourse worker

[Install]
WantedBy=multi-user.target

Create a postgres password for the concourse user:

cd /home/concourse/
sudo -u concourse psql atc
atc=> ALTER USER concourse WITH PASSWORD 'concourse';
atc=> \q

Start and Enable the Services:

systemctl start concourse-web concourse-worker
systemctl enable concourse-web concourse-worker postgresql
systemctl status concourse-web concourse-worker

systemctl is-active concourse-worker concourse-web
# active
# active

The listening ports should more or less look like the following:

netstat -tulpn

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:7777          0.0.0.0:*               LISTEN      4530/concourse
tcp        0      0 127.0.0.1:7788          0.0.0.0:*               LISTEN      4530/concourse
tcp        0      0 127.0.0.1:8079          0.0.0.0:*               LISTEN      4525/concourse
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1283/sshd
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      4047/postgres
tcp6       0      0 :::36159                :::*                    LISTEN      4525/concourse
tcp6       0      0 :::46829                :::*                    LISTEN      4525/concourse
tcp6       0      0 :::2222                 :::*                    LISTEN      4525/concourse
tcp6       0      0 :::8080                 :::*                    LISTEN      4525/concourse
tcp6       0      0 :::22                   :::*                    LISTEN      1283/sshd
udp        0      0 0.0.0.0:68              0.0.0.0:*                           918/dhclient
udp        0      0 0.0.0.0:42165           0.0.0.0:*                           4530/concourse

Client Side:

I will be using a the Fly cli from a Mac, so first we need to download the fly-cli for Mac:

wget https://github.com/concourse/concourse/releases/download/v4.2.2/fly_darwin_amd64
chmod +x fly_darwin_amd64
alias fly='./fly_darwin_amd64'

Next, we need to setup our Concourse Target by Authenticating against our Concourse Endpoint, lets setup our target with the name ci:

fly -t ci login -c http://10.20.30.40:8080
logging in to team 'main'

username: admin
password:

target saved

Lets list our targets:

fly targets
name  url                        team  expiry
ci    http://10.20.30.40:8080    main  Wed, 08 Nov 2017 15:32:59 UTC

Listing Registered Workers:

fly -t ci workers
name              containers  platform  tags  team  state    version
ip-172-31-12-134  0           linux     none  none  running  1.2

Listing Active Containers:

fly -t ci containers
handle                                worker            pipeline     job            build #  build id  type   name                  attempt

Hello World Pipeline:

Let's create a basic pipeline, that will print out Hello, World!:

Our hello-world.yml

jobs:
- name: my-job
  plan:
  - task: say-hello
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: alpine
          tag: edge
      run:
        path: /bin/sh
        args:
        - -c
        - |
          echo "============="
          echo "Hello, World!"
          echo "============="

Applying the configuration to our pipeline:

fly -t ci set-pipeline -p yeeehaa -c hello-world.yml
jobs:
  job my-job has been added:
    name: my-job
    plan:
    - task: say-hello
      config:
        platform: linux
        image_resource:
          type: docker-image
          source:
            repository: alpine
            tag: edge
        run:
          path: /bin/sh
          args:
          - -c
          - |
            echo "============="
            echo "Hello, World!"
            echo "============="

apply configuration? [yN]: y
pipeline created!
you can view your pipeline here: http://10.20.30.40:8080/teams/main/pipelines/yeeehaa

the pipeline is currently paused. to unpause, either:
  - run the unpause-pipeline command
  - click play next to the pipeline in the web ui

We can browse to the WebUI to unpause the pipeline, but since I like to do everything on cli as far as possible, I will unpause the pipeline via cli:

fly -t ci unpause-pipeline -p yeeehaa
unpaused 'yeeehaa'

Now our Pipeline is unpaused, but since we did not specify any triggers, we need to manually trigger the pipeline to run, you can either via the WebUI, select your pipeline which in this case will be named yeeehaa and then select the job, which will be my-job then hit the + sign, which will trigger the pipeline.

I will be using the cli:

fly -t ci trigger-job --job yeeehaa/my-job
started yeeehaa/my-job #1

Via the WebUI on http://10.20.30.40:8080/teams/main/pipelines/yeeehaa/jobs/my-job/builds/1 you should see the Hello, World! output, or via the cli, we also have the option to see the output, so let's trigger it again, but this time passing the --watch flag:

fly -t ci trigger-job --job yeeehaa/my-job --watch
started yeeehaa/my-job #2

initializing
running /bin/sh -c echo "============="
echo "Hello, World!"
echo "============="

=============
Hello, World!
=============
succeeded

Listing our Workers and Containers again:

fly -t ci workers
name              containers  platform  tags  team  state    version
ip-172-31-12-134  2           linux     none  none  running  1.2

fly -t ci containers
handle                                worker            pipeline     job         build #  build id  type   name           attempt
36982955-54fd-4c1b-57b8-216486c58db8  ip-172-31-12-134  yeeehaa      my-job      2        729       task   say-hello      n/a

Thank You

Thanks for reading, feel free to check out my website, feel free to subscribe to my newsletter or follow me at @ruanbekker on Twitter.