Introduction
If you have been using Gitlab and configured to run CI (Continuous Integration) using gitlab-ci.yml then you may not know that by default, it runs using the free runner provided by Gitlab
Project > Settings > CI/CD > Runner
You can choose to use the free shared runner provided by Gitlab or to run your own instance(s) of Gitlab-runner
Gitlab Architecture Overview
drawn using excalidraw
There needs to have at least one runner for the pipeline to run but you can scale it to as many runners as you like by just registering the instance to Gitlab
Whenever you push your code to the GitLab server, it looks for .gitlab-ci.yml and starts the pipeline by delegating the job to one of the runners. You can tag a runner, and configure your job to run on a specific runner
Setup your own runner
I tried to do this a couple of months back, but reading the documentation from Gitlab did not help as I absorbed the content more easily through visual examples (e.g youtube video, screenshots) and then I stumbled upon this video which cleared some of my initial doubts
Gitlab Runner can be installed on a variety of OS and means and it's actually very easy to do so now that I am able to. Hence, in this guide, I will try to answer some doubts I have previously, and what makes it all click together
Over the past few months, I have been using Gitlab and Gitlab pipeline and gained more knowledge and experience on the platform so I tried to set up my own runner again. The main reason is that I want to take advantage of the caching feature to reduce the time to run the pipeline job
Before I start
Here are some things that you should know
- You should know how to use
Docker and docker-compose
- Take note of the URL and token shown in introduction
Setup using docker-compose
I put together this docker-compose script to start my Gitlab-runner instance on my local machine. You are free to take (and modify) for own use
version: '3.7'
services:
gitlabee-runner:
image: gitlab/gitlab-runner:alpine
container_name: gitlabee-runner
volumes:
- gitlab_home_runner_config:/home/gitlab-runner
- gitlab_etc_runner_config:/etc/gitlab-runner
- /var/run/docker.sock:/var/run/docker.sock
restart: unless-stopped
volumes:
gitlab_home_runner_config:
gitlab_etc_runner_config:
Run docker-compose up -d and run docker ps and you should see the service running
❯ docker-compose up -d
❯ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a0e550df3149 gitlab/gitlab-runner:alpine "/usr/bin/dumb-init …" 13 seconds ago Up 5 seconds gitlabee-runner
Run docker logs and you should see the following
❯ docker logs gitlabee-runner
Runtime platform arch=amd64 os=linux pid=8 revision=5316d4ac version=14.6.0
Starting multi-runner from /etc/gitlab-runner/config.toml... builds=0
Running in system-mode.
Configuration loaded builds=0
listen_address not defined, metrics & debug endpoints disabled builds=0
[session_server].listen_address not defined, session endpoints disabled builds=0
ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory builds=0
ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory builds=0
ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory builds=0
ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory builds=0
Notice the ERROR, but not to worry, that is because we have not set up and registered our runner to any Gitlab server yet. The error will be gone once we have registered to Gitlab
Registration
Run docker exec -it gitlabee-runner bash
❯ docker exec -it gitlabee-runner bash
bash-5.0#
Let's register our runner, we will need the URL and token now
❯ docker exec -it gitlabee-runner bash
bash-5.0# [1] gitlab-runner register
Runtime platform arch=amd64 os=linux pid=27 revision=5316d4ac version=14.6.0
Running in system-mode.
Enter the GitLab instance URL (for example, https://gitlab.com/):
[2] https://gitlab.com/
Enter the registration token:
[3] yYrBvSeLyc4JxjGjmJxz
Enter a description for the runner:
[a0e550df3149]: [4] myrunner
Enter tags for the runner (comma-separated):
[5]
Registering runner... succeeded runner=yYrBvSeL
Enter an executor: shell, virtualbox, custom, docker, docker-ssh, parallels, ssh, docker+machine, docker-ssh+machine, kubernetes:
[6] docker
Enter the default Docker image (for example, ruby:2.6):
[7] alpine:3.15.0
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
bash-5.0# [8] exit
The numbering [1], [2], ... are inserted by me
- Execute the command to register the runner instance
- GitLab instance URL
- Registration token (note: your token should not be revealed to anyone, in my case, the token will be destroyed shortly after the post)
- Specify a name for your runner
- Specify one or more tags for your runner
- Tag can be configured in
.gitlab-ci.yml to specific which runner we want to execute the job on
docker should be the easiest in my opinion. Read more on executors in the documentation- The base image your job will run on when not specific within
.gitlab-ci.yml
- Exit the
shell
Default Docker image
Choosing the default docker image was confusing for me, and the example (ruby) stated there didn't help much. I wasn't sure when it's gonna be used, or how.
Take a look at any gitlab pipeline job log, and you will always see this (or similar) at the start
Running with gitlab-runner 14.6.0~beta.30.g4c96395a (4c96395a)
on blue-3.shared.runners-manager.gitlab.com/default zxwgkjAP
Preparing the "docker+machine" executor
00:30
Using Docker executor with image node:16.13.2 ...
Pulling docker image node:16.13.2 ...
Using docker image sha256:842962c4b3a745b6cfc84acf07688a392bcb90af27adb28a6b67534d9dccb79d for node:16.13.2 with digest node@sha256:4b0b5c3db44f567d5d25c80a6fe33a981d911cdae20b39d2395be268aea2cb97 ...
Notice that it is pulling node:16.13.2 image to execute my job, and that is what the default docker image is all about. And why is it node image and not alpine as specific during the setup above?
Let's take a look at the gitlab-ci.yml that I have defined. A full version is available here
image: node:16.13.2
stages:
- build
build website:
stage: build
script:
- npm i
- npm i -g gatsby-cli
- gatsby build
As mentioned in the docs, if we do not define an image to use within gitlab-ci.yml, the default base image will be used to run the job but we can override it by specifying the image we wish to use for the job as seen in the example above
Post Registration
Once registered, look at docker logs and you should see this
gitlabee-runner | Configuration loaded builds=0
gitlabee-runner | Runtime platform arch=amd64 os=linux pid=8 revision=5316d4ac version=14.6.0
gitlabee-runner | Starting multi-runner from /etc/gitlab-runner/config.toml... builds=0
Refresh the page in Gitlab and you should see Available specific runners on the left with the runner name that we specified earlier during the registration - myrunner
Let's turn off the shared runners for now by toggling Enable shared runners for this project. This would force all our pipelines to run on our runner instead of using the shared runners provided. Another way around it is to use tags to specific the runner if we want to force the pipeline to run on a specific runner
What's next?
Each Gitlab runner could be configured with specific use cases such as building java (Gradle) projects, nodejs projects, and so on.
This is so that end-user (developer) could just specify the tag for their job to run within gitlab-ci.yml and not have to care which exact runner their job runs on or the image it uses as long it does the job
Distributed Caching
If there is only a single runner instance, we can take advantage of the caching available and speed up the job which might be fine if it's just for yourself (single project) but that wouldn't be so in an organization where multiple runners are usually configured. Hence, we need to configure to store our cache externally so that the cache can be shared among the runners
Conclusion
Looking back, it is actually very easy to spin up your own Gitlab runner instance although that isn't necessary for a personal project but never hurts to know how you can do it, and it will definitely be useful if your organization is self-hosting its own Gitlab server
