Introduction

If you have been using Gitlab and configured to run CI (Continuous Integration) using gitlab-ci.yml then you may not know that by default, it runs using the free runner provided by Gitlab

Project > Settings > CI/CD > Runner

You can choose to use the free shared runner provided by Gitlab or to run your own instance(s) of Gitlab-runner

Gitlab Architecture Overview

drawn using excalidraw

There needs to have at least one runner for the pipeline to run but you can scale it to as many runners as you like by just registering the instance to Gitlab

Whenever you push your code to the GitLab server, it looks for .gitlab-ci.yml and starts the pipeline by delegating the job to one of the runners. You can tag a runner, and configure your job to run on a specific runner

Setup your own runner

I tried to do this a couple of months back, but reading the documentation from Gitlab did not help as I absorbed the content more easily through visual examples (e.g youtube video, screenshots) and then I stumbled upon this video which cleared some of my initial doubts

Gitlab Runner can be installed on a variety of OS and means and it's actually very easy to do so now that I am able to. Hence, in this guide, I will try to answer some doubts I have previously, and what makes it all click together

Over the past few months, I have been using Gitlab and Gitlab pipeline and gained more knowledge and experience on the platform so I tried to set up my own runner again. The main reason is that I want to take advantage of the caching feature to reduce the time to run the pipeline job

Before I start

Here are some things that you should know

You should know how to use Docker and docker-compose
- Otherwise, look for the installation guide on Gitlab for your choice of setup
Take note of the URL and token shown in introduction

Setup using docker-compose

I put together this docker-compose script to start my Gitlab-runner instance on my local machine. You are free to take (and modify) for own use

version: '3.7'

services:
  gitlabee-runner:
    image: gitlab/gitlab-runner:alpine
    container_name: gitlabee-runner
    volumes:
      - gitlab_home_runner_config:/home/gitlab-runner
      - gitlab_etc_runner_config:/etc/gitlab-runner
      # this is important as it needs to talk to your local docker daemon from within the container instance
      - /var/run/docker.sock:/var/run/docker.sock
    # This is not required to have but is something I do for all my service
    restart: unless-stopped

volumes:
  gitlab_home_runner_config:
    # external: true // Specify if you wish to create and manage yourself
  gitlab_etc_runner_config:
    # external: true // Specify if you wish to create and manage yourself

Run docker-compose up -d and run docker ps and you should see the service running

❯ docker-compose up -d
❯ docker ps
CONTAINER ID   IMAGE                          COMMAND                  CREATED          STATUS         PORTS                              NAMES
a0e550df3149   gitlab/gitlab-runner:alpine    "/usr/bin/dumb-init …"   13 seconds ago   Up 5 seconds                                      gitlabee-runner

Run docker logs and you should see the following

❯ docker logs gitlabee-runner
Runtime platform                                    arch=amd64 os=linux pid=8 revision=5316d4ac version=14.6.0
Starting multi-runner from /etc/gitlab-runner/config.toml...  builds=0
Running in system-mode.

Configuration loaded                                builds=0
listen_address not defined, metrics & debug endpoints disabled  builds=0
[session_server].listen_address not defined, session endpoints disabled  builds=0
ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory  builds=0
ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory  builds=0
ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory  builds=0
ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory  builds=0

Notice the ERROR, but not to worry, that is because we have not set up and registered our runner to any Gitlab server yet. The error will be gone once we have registered to Gitlab

Registration

Run docker exec -it gitlabee-runner bash

❯ docker exec -it gitlabee-runner bash
bash-5.0#

Let's register our runner, we will need the URL and token now

❯ docker exec -it gitlabee-runner bash
bash-5.0# [1] gitlab-runner register
Runtime platform                                    arch=amd64 os=linux pid=27 revision=5316d4ac version=14.6.0
Running in system-mode.

Enter the GitLab instance URL (for example, https://gitlab.com/):
[2] https://gitlab.com/
Enter the registration token:
[3] yYrBvSeLyc4JxjGjmJxz
Enter a description for the runner:
[a0e550df3149]: [4] myrunner
Enter tags for the runner (comma-separated):
[5] 
Registering runner... succeeded                     runner=yYrBvSeL
Enter an executor: shell, virtualbox, custom, docker, docker-ssh, parallels, ssh, docker+machine, docker-ssh+machine, kubernetes:
[6] docker
Enter the default Docker image (for example, ruby:2.6):
[7] alpine:3.15.0
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
bash-5.0# [8] exit

The numbering [1], [2], ... are inserted by me

Execute the command to register the runner instance
GitLab instance URL
Registration token (note: your token should not be revealed to anyone, in my case, the token will be destroyed shortly after the post)
Specify a name for your runner
Specify one or more tags for your runner
- Tag can be configured in .gitlab-ci.yml to specific which runner we want to execute the job on
docker should be the easiest in my opinion. Read more on executors in the documentation
The base image your job will run on when not specific within .gitlab-ci.yml
Exit the shell

Default Docker image

Choosing the default docker image was confusing for me, and the example (ruby) stated there didn't help much. I wasn't sure when it's gonna be used, or how.

Take a look at any gitlab pipeline job log, and you will always see this (or similar) at the start

Running with gitlab-runner 14.6.0~beta.30.g4c96395a (4c96395a)
  on blue-3.shared.runners-manager.gitlab.com/default zxwgkjAP
Preparing the "docker+machine" executor
00:30
Using Docker executor with image node:16.13.2 ...
Pulling docker image node:16.13.2 ...
Using docker image sha256:842962c4b3a745b6cfc84acf07688a392bcb90af27adb28a6b67534d9dccb79d for node:16.13.2 with digest node@sha256:4b0b5c3db44f567d5d25c80a6fe33a981d911cdae20b39d2395be268aea2cb97 ...

Notice that it is pulling node:16.13.2 image to execute my job, and that is what the default docker image is all about. And why is it node image and not alpine as specific during the setup above?

Let's take a look at the gitlab-ci.yml that I have defined. A full version is available here

# specify the base image to use unless overridden at the job level
image: node:16.13.2

stages:
  - build

build website:
  stage: build
  script:
    - npm i
    - npm i -g gatsby-cli
    - gatsby build

As mentioned in the docs, if we do not define an image to use within gitlab-ci.yml, the default base image will be used to run the job but we can override it by specifying the image we wish to use for the job as seen in the example above

Post Registration

Once registered, look at docker logs and you should see this

gitlabee-runner  | Configuration loaded                                builds=0
gitlabee-runner  | Runtime platform                                    arch=amd64 os=linux pid=8 revision=5316d4ac version=14.6.0
gitlabee-runner  | Starting multi-runner from /etc/gitlab-runner/config.toml...  builds=0

Refresh the page in Gitlab and you should see Available specific runners on the left with the runner name that we specified earlier during the registration - myrunner

Let's turn off the shared runners for now by toggling Enable shared runners for this project. This would force all our pipelines to run on our runner instead of using the shared runners provided. Another way around it is to use tags to specific the runner if we want to force the pipeline to run on a specific runner

What's next?

Configure your runner with tags

Each Gitlab runner could be configured with specific use cases such as building java (Gradle) projects, nodejs projects, and so on.

This is so that end-user (developer) could just specify the tag for their job to run within gitlab-ci.yml and not have to care which exact runner their job runs on or the image it uses as long it does the job

Distributed Caching

If there is only a single runner instance, we can take advantage of the caching available and speed up the job which might be fine if it's just for yourself (single project) but that wouldn't be so in an organization where multiple runners are usually configured. Hence, we need to configure to store our cache externally so that the cache can be shared among the runners

Conclusion

Looking back, it is actually very easy to spin up your own Gitlab runner instance although that isn't necessary for a personal project but never hurts to know how you can do it, and it will definitely be useful if your organization is self-hosting its own Gitlab server

Joseph Gan's Blog