cirruslabs / cirruslabs/gitlab-tart-executor

Error between runner-versions on pre-configured tart images.

Open
#143 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
102
Forks
14
Avg merge
21h 24m
Merged PRs (30d)
1

Description

`TART_EXECUTOR_INSTALL_GITLAB_RUNNER` is useful for ensuring a Tart VM has `gitlab-runner` available, but it appears to skip installation when any `gitlab-runner` binary is already present in the image.

That leaves a version mismatch unresolved when the Tart executor host runs a newer GitLab Runner than the guest image. In our case, newer host-side GitLab Runner versions generate internal helper commands that the older guest-side `gitlab-runner` does not understand.

The requested behavior is not necessarily a "force install" flag. A better fit would be version-aware reconciliation:

- If `TART_EXECUTOR_INSTALL_GITLAB_RUNNER` is set to a concrete version, install or update the guest binary when the existing version does not match.
- If it is set to `latest`, install or update when the guest binary is not the latest available version.
- If it is set to `brew`, run the Homebrew install or upgrade path rather than only checking that a binary exists.
- If it is set to `curl`, ensure the curl-installed binary is current according to the selected behavior.
- Skip only when the existing guest-side runner already satisfies the requested state.

## Environment

- `gitlab-tart-executor`: observed with `1.28.0`
- Host runner: GitLab Runner `18.10.0` or newer
- Guest image: Tart macOS image with an older preinstalled `gitlab-runner`
- Executor configuration uses `TART_EXECUTOR_INSTALL_GITLAB_RUNNER=`

## Problem

When the Tart image already contains `gitlab-runner`, the executor appears to skip installation even if `TART_EXECUTOR_INSTALL_GITLAB_RUNNER` requests a specific version.

This means the host and guest runner binaries can remain incompatible. The host runner generates the job scripts and internal commands, while the guest-side runner binary executes commands such as `gitlab-runner artifacts-uploader`.

Starting with GitLab Runner `18.10.0`, the generated artifact upload command can include `--timeout`. Older guest-side runner versions do not support that option.

## Error

```text
Incorrect Usage: flag provided but not defined: -timeout
NAME:
gitlab-runner artifacts-uploader - create and upload build artifacts (internal)
USAGE:
gitlab-runner artifacts-uploader [command options] [arguments...]
OPTIONS:
FATAL: flag provided but not defined: -timeout
--id value The build ID to download and upload artifacts for (default: "14914430309") [$CI_JOB_ID]
--token value Build token (default: "[MASKED]") [$CI_JOB_TOKEN]
--url value GitLab CI URL (default: "https://gitlab.com/") [$CI_SERVER_URL]
--tls-ca-file value File containing the certificates to verify the peer when using HTTPS (default: "/var/tmp/builds/strykercorp/robotics/geometry-modeling/meshlib.tmp/CI_SERVER_TLS_CA_FILE") [$CI_SERVER_TLS_CA_FILE]
--tls-cert-file value File containing certificate for TLS client auth with runner when using HTTPS [$CI_SERVER_TLS_CERT_FILE]
--tls-key-file value File containing private key for TLS client auth with runner when using HTTPS [$CI_SERVER_TLS_KEY_FILE]
--path value Add paths to archive (default: "[${OSEP_CPP_INSTALL_BASE_DIR}, ${OSEP_CPP_TEST_OUTPUT_BASE_DIR}]")
--exclude value Exclude paths from the archive
--untracked Add git untracked files
--verbose Detailed information
--transfer-meter-frequency value If set to more than 0s it enables an interactive transfer meter (default: "0s") [$TRANSFER_METER_FREQUENCY]
--generate-artifacts-metadata
--runner-id value (default: "0")
--repo-url value
--repo-digest value
--job-name value
--executor-name value
--runner-name value
--metadata-parameter value
--started-at value
--ended-at value
--schema-version value
--name value The name of the archive (default: "artifacts")
--expire-in value When to expire artifacts (default: "1 day")
--artifact-format value Format of generated artifacts (default: "zip")
--artifact-type value Type of generated artifacts (default: "archive")
--compression-level value Compression level (fastest, fast, default, slow, slowest) [$ARTIFACT_COMPRESSION_LEVEL]

Process exited with status 1
```

## Observed Behavior

1. The Tart guest image already contains `gitlab-runner`.
2. The executor starts the job.
3. `TART_EXECUTOR_INSTALL_GITLAB_RUNNER=` is set.
4. The executor detects that `gitlab-runner` exists and skips installation.
5. The older guest runner executes host-generated helper commands.
6. Artifact upload fails because the older guest runner does not support `--timeout`.

## Expected Behavior

When `TART_EXECUTOR_INSTALL_GITLAB_RUNNER` is configured, the executor should verify whether the existing guest runner satisfies the requested install mode.

For a concrete version:

```text
TART_EXECUTOR_INSTALL_GITLAB_RUNNER=18.9.0
```

The executor should run something equivalent to:

```sh
installed_version="$(gitlab-runner --version | awk '/Version:/ {print $2}')"
test "$installed_version" = "18.9.0" || install_gitlab_runner_18_9_0
```

For `latest`, the executor should compare the installed version to the latest available runner version and update if needed.

For `brew`, the executor should use Homebrew install or upgrade semantics.

The operation should remain idempotent: if the requested version is already installed, nothing should change.

## Why This Matters

The host runner and guest runner binaries are part of the same job execution path. A guest image can easily contain an older preinstalled runner, especially when Tart images are reused over time.

In this specific case, GitLab Runner `18.10.0` and newer can emit `artifacts-uploader --timeout`, while older guest-side runner versions fail with:

```text
flag provided but not defined: -timeout
```

Pinning the host runner to `18.9.0` works around the issue because `18.9.0` is the newest version we found before the artifact uploader `--timeout` option was introduced. That is only a workaround; the long-term fix is to keep the guest runner compatible with the host runner.

## Workaround

Current workaround options are:

- Pin the host GitLab Runner to `18.9.0`.
- Build Tart images without a preinstalled `gitlab-runner`, allowing `TART_EXECUTOR_INSTALL_GITLAB_RUNNER` to install it.
- Manually update the preinstalled guest runner in every Tart image.

The first option prevents use of newer GitLab Runner releases. The other two options move version reconciliation outside the executor, which makes image maintenance more fragile.

## Request

Please consider changing `TART_EXECUTOR_INSTALL_GITLAB_RUNNER` from an existence check to a version-aware reconciliation step.

The important behavior is:

- `major.minor.patch`: install that exact version when missing or mismatched.
- `latest`: install or update to the latest available version.
- `brew`: install or upgrade through Homebrew.
- `curl`: install or update through the curl-based installer.
- Existing matching versions should be left unchanged.

This would make Tart runner images safer to reuse and would avoid subtle host/guest GitLab Runner incompatibilities during artifact upload and other internal runner operations.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating where the executor handles TART_EXECUTOR_INSTALL_GITLAB_RUNNER and read the existing installation and skip logic. Check how it supports the requested version, latest, brew, and curl modes; done means it reconciles each mode with the installed guest runner and leaves an already-matching version unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
gitlab, go
Domain
devtools, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.