Local Gitlab runner with docker executor without Docker desktop
Awhile ago I had a CI/CD problem with one of my open-source projects because I regularly hit my CI/CD quota on Gitlab.com pipelines. The solution I came up with was to set up gitlab runner on my Macbook Pro, my daily-driver laptop. Like on Gitlab.com, I use a docker executor, and since I had Docker Desktop already installed, everything just worked. However, the laptop isn't an always-on machine, so I got a new base Mac mini and this post explains the setup. One of the goals was to not use Docker Desktop as the Mac mini has much lower RAM allotment (16 GB) than my laptop and is intended to be used headless.
The Initial local setup
There are two relevant configuration files:
| name | location |
|---|---|
| .gitlab-ci.yml | In the project's git repo |
| config.toml | ~/.gitlab-runner/config.toml |
-
.gitlab-ci.yml:variables: # When using dind service we need to instruct docker, to talk with the # daemon started inside the service. The daemon is available with # a network connection instead of the default /var/run/docker.sock socket. # # The 'docker' hostname is the alias of the service container as described at # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services # # Note that if you're using Kubernetes executor, the variable should be set to # tcp://localhost:2375 because of how Kubernetes executor connects services # to the job container DOCKER_HOST: tcp://docker:2375/ # When using dind, it's wise to use the overlayfs driver for # improved performance. DOCKER_DRIVER: overlay2 # See https://github.com/docker-library/docker/pull/166 DOCKER_TLS_CERTDIR: "" FF_NETWORK_PER_BUILD: "true" image: docker:latest services: - docker:dind -
config.toml
That setup is problematic for me though: When not in use, my laptop is closed, and occasionally it is disconnected from the Internet. That would be fine if I was the only one pushing merge requests, but that is not the case. Therefore, any MRs that are pushed against my project while the laptop is out-of commission, will see their pipelines fail.
Trying something new and problems encountered
I got a new Mac mini to use a homelab for self-hosting, not just for gitlab runner, but that's going to be the first use case.
I wanted to run gitlab runner in a distinct non-admin macOS user.
The Mac mini is the base model with small amount of memory (16 GB) so I was reluctant to use Docker Desktop (a memory-guzzling Electron app).
I recalled the colima project, which I understoo as a drop-in replacement for Docker Desktop.
Envisaged new setup:
On the new Mac, while logged-in as an admin user, I installed Homebrew and used it to install needed packages.
$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
$ brew install docker gitlab-runner colima
Info
docker here is a docker client used by us for testing and by gitlab-runner to talk to colima.
Then I set up gitlab-runner: After kicking off the registration process on my projects CI/CD settings on gitlab.com, I execute the interactive setup on command line:
Then I set up a new launchd per-user agent for the dedicated non-admin macOS user.
A quick sanity check with docker info reveals it cannot connect to a docker daemon as expected.
Then I start colima:
Rerunning docker info now works.
To test the setup, I triggered a pipeline for the project. It picked up the gitlab runner fine, but failed with an error:
Failure
ERROR: Preparation failed: getting docker info: failed to connect to the docker API at unix:///var/run/docker.sock; check if the path is correct and if the daemon is running: dial unix /var/run/docker.sock: connect: no such file or directory (docker.go:1383:0s)
By running colima status, I can see the reason:
$ colima status
INFO[0000] colima is running using macOS Virtualization.Framework
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] docker socket: unix:///Users/myuser/.colima/default/docker.sock
INFO[0000] containerd socket: unix:///Users/myuser/.colima/default/containerd.sock
The docker socket is written to the home directory of the user running colima (which is actually a good thing) but by default gitlab-runner expects it to be at the default system-wide location.
According to some hard to find Gitlab docs 1, there are three ways to solve it:
- Using an environment variable,
- Symlink colima socket to the location expected by the runner
- Specify colima socket in the runner configuration
I went for the configuration file option as it allows me to keep runner configuration in one place (which I can create a template for were I to scale this approach).
Additionally, the project Gitlab config needed to be modified to remove the DOCKER_HOST assignment.
Before I found the working solution above, I tried several things without success:
- Try running gitlab-runner as a system-wide service
- Try running gitlab-runner inside a container
- Try running colima as a system-wide service
When trying gitlab-runner inside a container, I got the error:
Failure
"docker: Error response from daemon: error while creating mount source path" source path (docker.socket) exist already
I didn't try to investigate further and pursue that approach, because I use Docker-in-docker in the pipeline jobs
and I foresaw I risk having to deal with nested virtualisation issues.
I don't know enough about colima to see this worth pursuing now.
At some other point, I was trying something (I can't remember what though) that got me a different error:
Failure
cannot access '/var/run/docker.sock': "Operation not supported"
Configurations changes
To summarise, here are the two configurations files from the beginning with the changes to make it work for the new setup:
.gitlab-ci.yml
variables:
# See https://github.com/docker-library/docker/pull/166
DOCKER_TLS_CERTDIR: ""
image: docker:latest
services:
- docker:dind
~/.gitlab-runner/config.toml
[[runners]]
executor = "docker"
environment = ["DOCKER_DRIVER=overlay2"]
[runners.feature_flags]
FF_ENABLE_JOB_CLEANUP = true
FF_NETWORK_PER_BUILD = true
[runners.docker]
tls_verify = false
image = "alpine:latest"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache"]
volume_keep = false
shm_size = 0
network_mtu = 0
host = "unix:///Users/myuser/.colima/default/docker.sock"
where myuser should be replaced by your chosen macOS non-admin user.
Persisting the setup
One last thing we need to do is to ensure colima restart if the machine is rebooted.
That issue is already solved for gitlab-runner when we ran gitlab-runner install,
as that created a launchd agent for our macOS non-admin user.
For colima, we could use brew services start colima, but it is a blackbox, and it has weird behaviour if
executed with a different user than the one that installed it.
Since I want both gitlab-runner and colima under the same non-admin user with a predictable means of communication between the two,
I have to manually create and start my own launchd agent.
The easiest way is to start from a template. When I installed colima,
amongst the files in the packages there is a .plist file.
That's the template we can leverage.
Info
A launchd agent template is installed by Homebrew alongside colima binary and other files.
Then create a colima.plist from that template and replace the label (with e.g: colima) and the various paths to match the directories for the chosen macOS non-admin user.
$ cp /opt/homebrew/Cellar/colima/0.10.3/homebrew.mxcl.colima.plist ~/Library/LaunchAgents/colima.plist
$ nvim ~/Library/LaunchAgents/colima.plist
$ launchctl load ~/Library/LaunchAgents/colima.plist
$ launchctl kickstart -k gui/<UID>/<label>
<UID> is the UID of the macOS user that should run those services,
and <label> is the label defined in colima.plist, under the <Label> property:
For the paths, change the following properties to match your current macOS user:
<key>StandardErrorPath</key>
<string>/Users/myuser/.local/logs/colima/colima.log</string>
<key>StandardOutPath</key>
<string>/Users/myuser/.local/logs/colima/colima.log</string>
<key>WorkingDirectory</key>
<string>/Users/myuser</string>
Finally, we can load and start the new service:
Success
To verify that all services are active, run:
$ launchctl list colima
{
"StandardOutPath" = "/Users/myuser/.local/logs/colima/colima.log";
"LimitLoadToSessionType" = "Aqua";
"StandardErrorPath" = "/Users/myuser/.local/logs/colima/colima.log";
"Label" = "colima";
"OnDemand" = true;
"LastExitStatus" = 0;
"Program" = "/opt/homebrew/opt/colima/bin/colima";
"ProgramArguments" = (
"/opt/homebrew/opt/colima/bin/colima";
"start";
"-f";
);
};
$ launchctl list gitlab-runner
{
"StandardOutPath" = "/Users/myuser/gitlab-runner.out.log";
"LimitLoadToSessionType" = "Aqua";
"StandardErrorPath" = "/Users/myuser/gitlab-runner.err.log";
"Label" = "gitlab-runner";
"OnDemand" = false;
"LastExitStatus" = 0;
"PID" = 78368;
"Program" = "/opt/homebrew/opt/gitlab-runner/bin/gitlab-runner";
"ProgramArguments" = (
"/opt/homebrew/opt/gitlab-runner/bin/gitlab-runner";
"run";
"--config";
"/Users/myuser/.gitlab-runner/config.toml";
"--working-directory";
"/Users/myuser";
"--service";
"gitlab-runner";
"--syslog";
);
};
Conclusion
It wasn't as turnkey as I'd hoped, but I got gitlab-runner working for my existing project's pipeline on a headless Mac, and without the need of Docker Desktop.
However, more work is needed for security hardening and automating the setup, so there may be more posts to follow this up in the future.