Skip to content

About

A GitLab Runner plugin for autoscaling servers on UpCloud.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

57 Commits

Folders and files

Repository files navigation

fleeting-plugin-upcloud

Latest Stable ntfy Release Channel Active Users Peak Users 30d Peak Users 365d

CI/CD Pipeline Status Coverage GitLab Last Commit GitLab Issues GitLab Merge Requests

A GitLab fleeting plugin that provisions ephemeral UpCloud servers as GitLab CI runner instances via the docker-autoscaler executor.

GitLab Runner runs on a small, long-lived runner manager host. The plugin creates UpCloud servers from a template you build once (with Docker preinstalled), the runner connects to them over SSH to run jobs in Docker, and the plugin deletes them again when they're no longer needed.

Prerequisites

  • A runner manager host (Linux, macOS or Windows) with GitLab Runner 16.11 or later installed. The manager doesn't need Docker. The CI instances it creates are always Linux servers on UpCloud.
  • An UpCloud API credential for the plugin. Either:
    • a Personal Access Token (recommended). Tokens only work from the IP ranges they allow, so allow the runner manager's address:

      upctl account token create --name gitlab-runner --expires-in 2160h --allow-ip-range "<runner manager IP>/32"

      Note the expiry, and create a new token before it runs out; or

    • an UpCloud username and password with API access enabled (a dedicated subaccount is best).

  • The runner manager must be able to reach the instances over SSH (port 22) — on their public IPv4 address, or on a private network with use_private_network.
  • To build the instance template: upctl and jq. Both are installed by mise install if you work from a clone of this repository.

Installation and upgrading

The plugin runs wherever GitLab Runner runs. Follow the guide for your runner manager's OS. Each one covers installing the plugin, the SSH key, registering the runner, the OS-specific config lines, restarting, upgrading and rolling back:

Runner manager OS Guide Installer
Linux (amd64, arm64) docs/install-linux.md install-plugin.sh
macOS (Apple Silicon, Intel) docs/install-macos.md install-plugin.sh
Windows (amd64, arm64) docs/install-windows.md install-plugin.ps1

Quick reference:

# Linux / macOS
curl -fsSL https://gitlab.com/KirboDev/agentic-coding/gitlab-fleeting-plugin-upcloud/-/raw/main/scripts/install-plugin.sh | bash
# Windows (elevated PowerShell)
irm https://gitlab.com/KirboDev/agentic-coding/gitlab-fleeting-plugin-upcloud/-/raw/main/scripts/install-plugin.ps1 | iex

Both installers verify the release checksum, install the latest release unless VERSION is set (e.g. v1.4.0), and can replace the binary while the runner is running. The runner only loads a new version when it restarts. Check the release notes for config changes before upgrading. To hear about new releases, see Update notifications.

GitLab Runner finds the plugin through the plugin setting. That's either the bare binary name fleeting-plugin-upcloud, looked up on the runner's PATH, or the full path to the binary. The guides say which to use on each OS.

Creating a custom UpCloud template

Using a pre-baked template (a server image that already has Docker installed) means instances are ready in seconds rather than having to bootstrap from scratch on every boot. Per GitLab's docs, instances need Docker but not GitLab Runner, and must never register themselves as runners; the image script follows that.

Run these steps from any machine with upctl and jq (authenticated to your UpCloud account).

1. Create a builder server based on Debian 13 Trixie

upctl server create \
  --zone fi-hel1 \
  --os "Debian GNU/Linux 13 (Trixie)" \
  --os-storage-size 10 \
  --hostname "gitlab-runner-template" \
  --plan "4xCPU-8GB" \
  --network family=IPv4,type=public \
  --ssh-keys ~/.ssh/id_ed25519.pub \
  --wait

--ssh-keys is your own public key, used only to log in to the builder. --wait returns once the server has started. If the OS name has changed, list the available ones with upctl storage list --public --template.

2. Prepare the server

TEMPLATE_IP="$(upctl server show gitlab-runner-template -o json \
  | jq -r '.ip_addresses[] | select(.access == "public" and .family == "IPv4") | .address')"

ssh "root@${TEMPLATE_IP}" \
  "curl -fsSL 'https://gitlab.com/KirboDev/agentic-coding/gitlab-fleeting-plugin-upcloud/-/raw/main/scripts/custom-image-debian13.sh' | bash -s -- --all"

This installs Docker and cloud-init and then cleans the server for imaging. It removes SSH host keys, authorized_keys, logs, machine-id and cloud-init state. If SSH isn't accepting connections yet, wait a few seconds and retry. Replace main in the URL with a release tag to pin the script version.

3. Stop the server

upctl server stop --wait gitlab-runner-template

Only continue if the previous step succeeded: a template made from a half-prepared server will produce broken instances.

4. Templatise the storage

upctl storage templatise "$(upctl server show gitlab-runner-template -o json | jq -r '.storage_devices[0].storage')" --title "GitLab Runner - Debian 13 ($(date +%F))" --wait

This creates a new private template in your account without touching the original server.

The $(date +%F) suffix date-stamps the title (e.g. GitLab Runner - Debian 13 (2026-06-08)). Each templatise mints a new template UUID, so when you rebuild later to pick up package updates the date in the title lets you tell versions apart in upctl storage list --template and confirm which one your config points at.

5. Delete the builder server and its original storage

upctl server delete --delete-storages gitlab-runner-template

6. Retrieve the template UUID

upctl storage list --template

Copy the UUID — this is the value to use as template in [runners.autoscaler.plugin_config]. When you rebuild the template later, update template and restart the runner.

Configuration

Edit the runner's config.toml: /etc/gitlab-runner/config.toml on Linux, ~/.gitlab-runner/config.toml on macOS, C:\GitLab-Runner\config.toml on Windows. Keep the fields gitlab-runner register wrote (id, token, token_obtained_at, …) and add the autoscaler sections. The example uses Linux values for plugin and key_path; your OS guide has the right ones for macOS and Windows.

# At most max_instances × capacity_per_instance jobs can run at once.
concurrent = 20
# Seconds the runner gives plugins to clean up on shutdown (0 = default of 30,
# which is tight for deleting instances).
shutdown_timeout = 120

[[runners]]
  name = "UpCloud GitLab Runner"
  url = "https://gitlab.com"
  token = "<filled in by gitlab-runner register>"
  executor = "docker-autoscaler"
  request_concurrency = 2          # default 1 delays jobs while GitLab long-polls

  [runners.docker]
    image = "alpine:latest"

  [runners.autoscaler]
    plugin = "fleeting-plugin-upcloud"
    capacity_per_instance = 4      # jobs running on one instance at the same time
    max_use_count = 60             # jobs per instance before it is replaced
    max_instances = 5
    instance_ready_command = "docker info"
    delete_instances_on_shutdown = true

    [runners.autoscaler.plugin_config]
      # Auth: use a Personal Access Token (recommended) or username + password
      token = "<your UpCloud API token>"
      # username = "<your UpCloud username>"
      # password = "<your UpCloud password>"
      template = "<your template UUID>"
      name = "my-runner-group"     # unique per runner manager, see below
      plan = "4xCPU-8GB"
      storage_size = 20
      storage_tier = "maxiops"
      zone = "fi-hel1"

    [runners.autoscaler.connector_config]
      username = "root"
      key_path = "/root/.ssh/fleeting-upcloud"
      use_external_addr = true

    [[runners.autoscaler.policy]]
      idle_count = 0               # jobs' worth of idle capacity to keep ready
      idle_time = "45m0s"          # how long an idle instance is kept

config.toml contains the UpCloud credential, so keep it readable only by the account the runner runs as. Restart the runner after editing it, as described in your OS guide.

Settings left out of the example (update_interval, deletion_retry_interval, failure_threshold, …) use GitLab Runner's defaults. See the [runners.autoscaler] reference.

Isolation trade-off: with capacity_per_instance and max_use_count above 1, jobs share instances (and their Docker daemon) with other jobs, which is faster and cheaper. If jobs from different projects or users must not affect each other, set both to 1 so that each job gets a fresh instance, as GitLab's examples do.

Plugin config reference

All fields go under [runners.autoscaler.plugin_config].

Field Required Default Description
token yes* — UpCloud Personal Access Token (ucat_…)
username yes* — UpCloud API username (alternative to token)
password yes* — UpCloud API password (required with username)
zone yes — UpCloud zone, e.g. fi-hel1
template yes — UpCloud template UUID to clone for each instance. Checked at startup.
name yes — Group name, stored on every server as the fleeting-group label. It identifies which servers the plugin manages, so it must be unique per runner manager: two managers using the same name would delete each other's servers. GitLab's docs also warn against sharing autoscaling resources between runner managers.
plan no 1xCPU-2GB UpCloud server plan (upctl server plans)
storage_tier no (from template) maxiops, standard or hdd
storage_size no (from template) Storage size in GB
name_prefix no fleeting Prefix for generated hostnames. Sanitized to a valid RFC 1123 label: lowercased, with invalid characters collapsed to hyphens (e.g. CI → ci). The server title keeps its original casing via title_prefix.
title_prefix no fleeting-plugin-upcloud Prefix for generated server titles
max_size no 100 Maximum number of instances the plugin allows. max_instances must not exceed it.
network no — UUID of an UpCloud private network (upctl network list) to attach to every instance, in addition to the public interface. Required by use_private_network.
use_private_network no false Connect to instances over their private IP on network instead of the public IP. The runner manager must be attached to the same network.
user_data no — URL or inline script for cloud-init on first boot
telemetry no true Send anonymous usage pings; set to false to opt out. See Telemetry.
update_check no true Check for newer releases at startup and hourly, and log a warning when one exists. See Update notifications.
update_notify_url no — Your own ntfy topic URL, e.g. https://ntfy.sh/<your topic>. The plugin posts there once per new release. Requires update_check.
update_notify_token no — ntfy access token (tk_…) for a topic that needs one. Requires update_notify_url.

* Either token or both username+password must be provided.

How it works

On each autoscaler cycle the plugin:

  1. Update — lists all UpCloud servers labelled with the group name and reports their state to the runner. Servers that are stopped or in the error state can't run jobs but are still billed for storage, so the plugin deletes them in the background. This covers servers it is removing itself, servers whose earlier delete failed, and servers stopped by anything else. If you stop a group server by hand, for example to debug it, it will be deleted.
  2. Increase — clones the template to create new servers, injecting the SSH public key derived from connector_config.key_path.
  3. Decrease — hard-stops instances that are no longer needed, without waiting for them to stop, so scaling isn't held up. Update deletes each one once it has stopped.
  4. ConnectInfo — returns the public (or private) IPv4 address and SSH details so the runner can connect.
  5. Heartbeat — reports instances that are gone, stopped or in the error state as unhealthy, so the runner replaces them.
  6. Shutdown — before the runner exits, waits for removals still in progress, so stopped servers aren't left behind.

Telemetry

Every ~15 minutes while the runner is running, the plugin sends an anonymous ping to a self-hosted Aptabase instance (aptabase.devaus.eu). The ping contains:

  • the plugin version and release channel (stable, or beta for prereleases);
  • the runner manager's OS and CPU architecture;
  • the configured UpCloud zone and plan;
  • a stable per-account id: an HMAC-SHA256 of your UpCloud account username, keyed with your API token (or password). Neither the username nor any credential is sent, and because the key never leaves the runner manager, the id can't be matched to your username even by someone who knows it.

Aptabase also derives an approximate country from the request IP. Nothing about your jobs, projects, servers or GitLab instance is sent. Development builds never send pings, and failed pings are only logged at debug level, so they never affect the runner.

The id lets the same install be counted once over a rolling ~24h window. All runner groups that use the same UpCloud account and credential count as one user. Creating a new token gives a new id. This is a personal project maintained in my free time, and knowing how many people use it helps me decide how much effort to put into it.

To opt out, add telemetry = false under [runners.autoscaler.plugin_config] and restart the runner.

Update notifications

The plugin checks GitLab for a newer release when it starts and then every hour. If it finds one, it logs a warning in the runner's log, once per release, like this:

a newer release of the plugin is available  current=v1.4.0 latest=v1.5.0 url=https://gitlab.com/…/-/releases/v1.5.0

Stable builds only compare against stable releases. Prerelease builds (e.g. 1.5.0-rc.1) also consider newer prereleases. The check only reads GitLab's public releases API and sends nothing but the request itself. Development builds never check, and failed checks are only logged at debug level. To turn it off, set update_check = false.

To get a push notification instead of reading logs, use ntfy in one of two ways:

  • Subscribe to the release channel, ntfy.sh/OKjnFezDgjw7JE6B, in the ntfy app or web app. The release pipeline posts each new release there once, with its release notes. Nothing to configure in the plugin.

  • Have the plugin post to your own topic: pick a hard-to-guess topic name (anyone who knows it can read and post to it), subscribe to it, and set it as update_notify_url:

    [runners.autoscaler.plugin_config]
      # ...
      update_notify_url = "https://ntfy.sh/<your topic>"
      # update_notify_token = "tk_..."  # only for a topic that needs an access token

    The plugin posts once per new release for as long as it runs: after a restart, it posts again if it's still out of date. Each message names the runner group (name) that's out of date. Each [[runners]] entry runs its own plugin, so every group that uses the topic posts once. Self-hosted ntfy servers work too. A failed post is logged as a warning and retried on the next check.

The release channel isn't accepted as update_notify_url: if every install posted there, subscribers would get one copy per install.

Troubleshooting

  • Logs: plugin messages appear in the runner's log: sudo journalctl -u gitlab-runner -f on Linux, the Event Viewer's Application log on Windows, and see the macOS guide for macOS.
  • fleeting list doesn't show the plugin: the runner can't find the binary. On Linux, make sure sudo which fleeting-plugin-upcloud finds it. On macOS and Windows, use the full path in plugin.
  • authenticating with UpCloud API at startup: the credentials are wrong or expired, or the request comes from an IP the token (or the account's API access settings) doesn't allow.
  • template … not found at startup: template doesn't match a storage UUID in your account. Check upctl storage list --template.
  • Instances are created but jobs never start: the runner can't reach them over SSH. Check that port 22 is reachable from the runner manager, that key_path and username are correct, and that the template has cloud-init (the image script installs it).
  • plugin process exited error=exit status 2 when the runner stops, followed by decreasing instances errors: systemd killed the plugins along with the runner. Add KillMode=mixed as described in the Linux guide. Instances left behind this way are removed after the next start.
  • Leftover servers: every server the plugin creates is titled <title_prefix> - <hostname> and labelled fleeting-group=<name> (shown by upctl server show <uuid>). List them by title with upctl server list -o json | jq -r '.servers[] | select(.title | startswith("fleeting-plugin-upcloud")) | "\(.uuid) \(.state) \(.title)"'.

Development

The toolchain (Go, golangci-lint, shellcheck, gotestsum, plus jq and upctl for the template workflow) and the tasks are managed with mise and defined in mise.toml. Install mise, then from the repository root:

mise trust   # approve this repo's mise.toml (first time only)
mise install
mise run build        # build for the current machine into bin/ (macOS: also ad-hoc signs it)
mise run dist         # build every release binary with .sha256 files into dist/, as CI does
mise run check        # vet, golangci-lint, shellcheck, race-enabled tests, govulncheck
mise tasks            # list all tasks (build-all, install, deploy, clean, …)

mise run test-integration runs fleeting's conformance test against real, billed UpCloud servers (up to 3 servers, deleted afterwards). It needs UPCLOUD_TOKEN, UPCLOUD_ZONE, UPCLOUD_TEMPLATE and UPCLOUD_SSH_KEY; see integration_test.go.

CI/CD

The pipeline runs only the jobs whose files a push or merge request changes. A change to only docs (*.md, docs/, LICENSE) or renovate.json starts no pipeline at all.

Changed Jobs
Go code, go.mod/go.sum, mise.toml, the build scripts vet, golangci-lint, race-enabled tests (results and coverage appear in merge requests), govulncheck, the build, and on main the release
Shell scripts shellcheck; for install-plugin.sh also verify-installer, which installs the latest release with the changed script
PowerShell scripts PSScriptAnalyzer
telemetry/, fnox.toml, ci/secrets.sh (on main) deploy-telemetry, which deploys the telemetry badge backend
Anything but docs and tool config GitLab Secret Detection
.gitlab-ci.yml everything

The release jobs, on main when the binary changes:

  1. Build: every release binary, with checksums, checking that the binary for the build host runs and reports the right version. Also runs on merge requests, so a cross-compile error shows up before it reaches main.
  2. Release: uploads the binaries and creates the GitLab release. This only happens after every target has built, so a release is never published half-built.
  3. Verify: installs the new release with install-plugin.sh, both pinned and as "latest", and checks the version it reports. refresh-telemetry-badges then regenerates the version badges, and notify-release posts the release to the ntfy release channel.

A new push cancels the checks still running for the previous one; release and deploy jobs are never cancelled. A pipeline started from the web UI runs every job.

Jobs run on a self-hosted Mac mini runner (tag macos, a Docker executor running linux/arm64 containers). While the Mac is offline, jobs wait for it. No job uses the upcloud runner, which runs on this plugin, so a broken release can't block the pipeline that fixes it.

Releases are automated: pushes to main with releasable Conventional Commits (feat:, fix:, …) produce a new SemVer version: a breaking change (type!: or a BREAKING CHANGE: footer) bumps the major version, feat: the minor, fix: and perf: the patch. The version and release notes come from git-cliff (cliff.toml, scripts/next-version.sh); mise run next-version previews the next release locally. Other types (chore:, ci:, build:, docs:, …) don't trigger a release, so dependency updates that end up in the binary should use fix(deps):. Renovate is configured to do that. The release jobs only run when the binary changes, so a releasable commit that touches only other files (e.g. a fix(scripts):) is released together with the next change to the binary.

One-time project settings worth enabling:

  • A weekly pipeline schedule on main. It runs only govulncheck, so newly disclosed vulnerabilities show up without a push. It never builds or releases.
  • Integration test: to get a manual integration job, add UPCLOUD_TOKEN (masked), UPCLOUD_ZONE and UPCLOUD_TEMPLATE as CI/CD variables, and UPCLOUD_SSH_KEY as a File variable holding a private key. Use protected variables if only main should run it.
  • Telemetry backend: the deploy jobs read their secrets from fnox.toml with the identity GitLab gives environment release/deploy. Set up the keys and values once as described in docs/SECRETS.md. To redeploy without a change to its files, run a pipeline on main from the web UI.

Contributing

Issues and merge requests are welcome at gitlab.com/KirboDev/agentic-coding/gitlab-fleeting-plugin-upcloud.

License

MIT

About

A GitLab Runner plugin for autoscaling servers on UpCloud.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages