# How to setup and configure your own Gitlab Runner # Introduction If you have been using [Gitlab](https://gitlab.com/) and configured to run [CI](https://docs.gitlab.com/ee/ci/) (Continuous Integration) using [gitlab-ci.yml](https://docs.gitlab.com/ee/ci/yaml/gitlab_ci_yaml.html) then you may not know that by default, it runs using the free runner provided by Gitlab > Project > Settings > CI/CD > Runner ![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1642222369272/Ndcn4YPPu.png) You can choose to use the free shared runner provided by `Gitlab` or to run your own instance(s) of `Gitlab-runner` # Gitlab Architecture Overview ![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1642222770589/bHOIymQHw.png) *drawn using [excalidraw](https://excalidraw.com/)* There needs to have at least one runner for the pipeline to run but you can scale it to as many runners as you like by just registering the instance to `Gitlab` Whenever you push your code to the GitLab server, it looks for `.gitlab-ci.yml` and starts the pipeline by delegating the job to one of the runners. You can [tag](https://docs.gitlab.com/ee/ci/runners/configure_runners.html#use-tags-to-control-which-jobs-a-runner-can-run) a runner, and configure your job to run on a specific runner # Setup your own runner I tried to do this a couple of months back, but reading the documentation from Gitlab did not help as I absorbed the content more easily through visual examples (e.g youtube video, screenshots) and then I stumbled upon this [video](https://www.youtube.com/watch?v=G8ZONHOTAQk) which cleared some of my initial doubts Gitlab Runner can be [installed](https://docs.gitlab.com/runner/install/index.html) on a variety of OS and means and it's actually very easy to do so now that I am able to. Hence, in this guide, I will try to answer some doubts I have previously, and what makes it all click together Over the past few months, I have been using Gitlab and Gitlab pipeline and gained more knowledge and experience on the platform so I tried to set up my own runner again. The main reason is that I want to take advantage of the caching feature to reduce the time to run the pipeline job ## Before I start Here are some things that you should know - You should know how to use `Docker` and `docker-compose` - Otherwise, look for the [installation guide](https://docs.gitlab.com/runner/install/index.html) on Gitlab for your choice of setup - Take note of the URL and token shown in [introduction](#introduction) ## Setup using docker-compose I put together this `docker-compose` script to start my `Gitlab-runner` instance on my local machine. You are free to take (and modify) for own use ```yml version: '3.7' services: gitlabee-runner: image: gitlab/gitlab-runner:alpine container_name: gitlabee-runner volumes: - gitlab_home_runner_config:/home/gitlab-runner - gitlab_etc_runner_config:/etc/gitlab-runner # this is important as it needs to talk to your local docker daemon from within the container instance - /var/run/docker.sock:/var/run/docker.sock # This is not required to have but is something I do for all my service restart: unless-stopped volumes: gitlab_home_runner_config: # external: true // Specify if you wish to create and manage yourself gitlab_etc_runner_config: # external: true // Specify if you wish to create and manage yourself ``` Run `docker-compose up -d` and run `docker ps` and you should see the service running ```log ❯ docker-compose up -d ❯ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a0e550df3149 gitlab/gitlab-runner:alpine "/usr/bin/dumb-init …" 13 seconds ago Up 5 seconds gitlabee-runner ``` Run `docker logs ` and you should see the following ```log ❯ docker logs gitlabee-runner Runtime platform arch=amd64 os=linux pid=8 revision=5316d4ac version=14.6.0 Starting multi-runner from /etc/gitlab-runner/config.toml... builds=0 Running in system-mode. Configuration loaded builds=0 listen_address not defined, metrics & debug endpoints disabled builds=0 [session_server].listen_address not defined, session endpoints disabled builds=0 ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory builds=0 ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory builds=0 ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory builds=0 ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such file or directory builds=0 ``` Notice the `ERROR`, but not to worry, that is because we have not set up and registered our runner to any Gitlab server yet. The error will be gone once we have registered to `Gitlab` ### Registration Run `docker exec -it gitlabee-runner bash` ```log ❯ docker exec -it gitlabee-runner bash bash-5.0# ``` Let's [register](https://docs.gitlab.com/runner/register/) our runner, we will need the `URL` and `token` now ```log ❯ docker exec -it gitlabee-runner bash bash-5.0# [1] gitlab-runner register Runtime platform arch=amd64 os=linux pid=27 revision=5316d4ac version=14.6.0 Running in system-mode. Enter the GitLab instance URL (for example, https://gitlab.com/): [2] https://gitlab.com/ Enter the registration token: [3] yYrBvSeLyc4JxjGjmJxz Enter a description for the runner: [a0e550df3149]: [4] myrunner Enter tags for the runner (comma-separated): [5] Registering runner... succeeded runner=yYrBvSeL Enter an executor: shell, virtualbox, custom, docker, docker-ssh, parallels, ssh, docker+machine, docker-ssh+machine, kubernetes: [6] docker Enter the default Docker image (for example, ruby:2.6): [7] alpine:3.15.0 Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded! bash-5.0# [8] exit ``` > The numbering [1], [2], ... are inserted by me 1. Execute the command to register the runner instance 2. GitLab instance URL 3. Registration token (note: your token should not be revealed to anyone, in my case, the token will be destroyed shortly after the post) 4. Specify a name for your runner 5. Specify one or more tags for your runner - Tag can be configured in `.gitlab-ci.yml` to specific which runner we want to execute the job on 6. `docker` should be the easiest in my opinion. Read more on [executors](https://docs.gitlab.com/runner/executors/) in the documentation 7. The base image your job will run on when not specific within `.gitlab-ci.yml` 8. Exit the `shell` #### Default Docker image Choosing the default docker image was confusing for me, and the example (ruby) stated there didn't help much. I wasn't sure when it's gonna be used, or how. Take a look at any [gitlab pipeline job log](https://gitlab.com/bwgjoseph/gatsby-static-site/-/jobs/1978319460), and you will always see this (or similar) at the start ```log Running with gitlab-runner 14.6.0~beta.30.g4c96395a (4c96395a) on blue-3.shared.runners-manager.gitlab.com/default zxwgkjAP Preparing the "docker+machine" executor 00:30 Using Docker executor with image node:16.13.2 ... Pulling docker image node:16.13.2 ... Using docker image sha256:842962c4b3a745b6cfc84acf07688a392bcb90af27adb28a6b67534d9dccb79d for node:16.13.2 with digest node@sha256:4b0b5c3db44f567d5d25c80a6fe33a981d911cdae20b39d2395be268aea2cb97 ... ``` Notice that it is pulling `node:16.13.2` image to execute my job, and that is what the default docker image is all about. And why is it `node` image and not `alpine` as specific during the setup above? Let's take a look at the `gitlab-ci.yml` that I have defined. A full version is available [here](https://gitlab.com/bwgjoseph/gatsby-static-site/-/blob/main/.gitlab-ci.yml) ```yml # specify the base image to use unless overridden at the job level image: node:16.13.2 stages: - build build website: stage: build script: - npm i - npm i -g gatsby-cli - gatsby build ``` As mentioned in the [docs](https://docs.gitlab.com/ee/ci/yaml/#image), if we do not define an image to use within `gitlab-ci.yml`, the default base image will be used to run the job but we can override it by specifying the image we wish to use for the job as seen in the example above ### Post Registration Once registered, look at `docker logs` and you should see this ```log gitlabee-runner | Configuration loaded builds=0 gitlabee-runner | Runtime platform arch=amd64 os=linux pid=8 revision=5316d4ac version=14.6.0 gitlabee-runner | Starting multi-runner from /etc/gitlab-runner/config.toml... builds=0 ``` Refresh the page in `Gitlab` and you should see `Available specific runners` on the left with the runner name that we specified earlier during the registration - `myrunner` ![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1642310695504/uanikdxCy.png) Let's turn off the shared runners for now by toggling `Enable shared runners for this project`. This would force all our pipelines to run on our runner instead of using the shared runners provided. Another way around it is to use `tags` to specific the runner if we want to force the pipeline to run on a specific runner # What's next? ### Configure your runner with tags Each Gitlab runner could be configured with specific use cases such as building java (Gradle) projects, nodejs projects, and so on. This is so that end-user (developer) could just specify the tag for their job to run within `gitlab-ci.yml` and not have to care which exact runner their job runs on or the image it uses as long it does the job ### Distributed Caching If there is only a single runner instance, we can take advantage of the caching available and speed up the job which might be fine if it's just for yourself (single project) but that wouldn't be so in an organization where multiple runners are usually configured. Hence, we need to configure to [store our cache](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) externally so that the cache can be shared among the runners # Conclusion Looking back, it is actually very easy to spin up your own Gitlab runner instance although that isn't necessary for a personal project but never hurts to know how you can do it, and it will definitely be useful if your organization is self-hosting its own Gitlab server