cirruslabs / cirruslabs/gitlab-tart-executor

CUSTOM_ENV_* requirements breaking custom executor with GitLab Runner

Open
#140 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
102
Forks
14
Avg merge
21h 24m
Merged PRs (30d)
1

Description

### Context

I’m having trouble using `gitlab-tart-executor` as a custom executor with GitLab Runner on macOS (local development machine). After several attempts with different Runner and Tart versions, the blocker clearly appears inside `gitlab-tart-executor` itself, at the level of the `CUSTOM_ENV_*` environment handling.

The goal is to use CirrusLabs macOS images (Sequoia/Tahoe) in GitLab CI via `gitlab-tart-executor` on a macOS Apple Silicon machine (M series).

### Observed behavior

1. The executor works correctly for the `--version` command:

```bash
/opt/homebrew/bin/gitlab-tart-executor --version
executor version 1.28.0-6e163f8
```

2. However, even a simple `config` fails immediately when `CUSTOM_ENV_CI_JOB_ID` is not present in the environment, including outside of GitLab Runner:

```bash
/opt/homebrew/bin/gitlab-tart-executor config
GitLab environment error: CUSTOM_ENV_CI_JOB_ID is missing
```

3. In GitLab CI, with the Runner configured in `executor = "custom"` and calling `gitlab-tart-executor` directly for `config_exec` / `prepare_exec` / `run_exec` / `cleanup_exec`, the job always fails with:

```text
Preparing the "custom" executor
Using Custom executor with driver tart 1.28.0-6e163f8...
"prepare" stage failed: CUSTOM_ENV_CI_JOB_ID is missing and no --default-image was set
WARNING: Cleanup script failed: exit status 2
ERROR: Job failed: exit status 1
```

4. In the runner logs, you can see the job being picked up and then going straight to cleanup, without any visible Tart VM creation:

```text
Checking for jobs... received ...
Added job to processing list ...
WARNING: Failed to stop VM: tart command returned non-zero exit code: "the specified VM \"gitlab-\" does not exist"
WARNING: Failed to delete VM: VM errored: failed to delete VM gitlab-: tart command returned non-zero exit code: "the specified VM \"gitlab-\" does not exist"
WARNING: VM errored: failed to delete VM gitlab-: tart command returned non-zero exit code: "the specified VM \"gitlab-\" does not exist"
WARNING: Cleanup script failed: exit status 2
WARNING: Job failed: exit status 1
```

In other words, `gitlab-tart-executor` refuses to run `config`/`prepare` as long as `CUSTOM_ENV_CI_JOB_ID` (and potentially other `CUSTOM_ENV_*` variables) are not defined. The error message also mentions that no `--default-image` was set.

### GitLab Runner configuration (excerpt)

Here is an example of my current runner configuration:

```toml
concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
session_timeout = 1800

[[runners]]
name = "MacBook-M1Max-Tart"
url = "https://gitlab.com"
id = 52312176
token = "REDACTED"
executor = "custom"
environment = ["TART_EXECUTOR_SOFTNET=true", "TART_EXECUTOR_ALWAYS_PULL=false"]

[runners.feature_flags]
FF_RESOLVE_FULL_TLS_CHAIN = false

[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]

[runners.custom]
config_exec = "/opt/homebrew/bin/gitlab-tart-executor"
config_args = ["config"]

prepare_exec = "sudo"
prepare_args = [
"/opt/homebrew/bin/gitlab-tart-executor", "prepare",
"--user", "juliengoudet",
"--concurrency", "1",
"--cpu", "auto",
"--memory", "auto"
]

run_exec = "sudo"
run_args = [
"/opt/homebrew/bin/gitlab-tart-executor", "run",
"--user", "juliengoudet"
]

cleanup_exec = "/opt/homebrew/bin/gitlab-tart-executor"
cleanup_args = ["cleanup"]
```

The Tart CLI itself (`tart`) works fine when used directly to launch CirrusLabs macOS images on this machine.

### What I already tried

- GitLab Runner 18.8.0 / 18.9.0: same error in CI
`"prepare" stage failed: CUSTOM_ENV_CI_JOB_ID is missing and no --default-image was set`.
- Downgraded GitLab Runner to 17.11.3: same behavior, i.e. `gitlab-tart-executor config` and `prepare` refuse to start without `CUSTOM_ENV_CI_JOB_ID` being defined.
- Verified `sudoers` and permissions:
`sudo /opt/homebrew/bin/gitlab-tart-executor --version` works without a password, but `config` still fails with the same `CUSTOM_ENV_*` error.
- Manual tests with `CUSTOM_ENV_CI_JOB_ID` defined in the environment before calling `gitlab-tart-executor config`: in that case, `config` starts working, which confirms that the hard requirement on `CUSTOM_ENV_*` is enforced inside the executor.

### Problem / questions

From what I can see:

- `gitlab-tart-executor` 1.28.0-6e163f8 is designed to be invoked in an environment where GitLab provides the `CUSTOM_ENV_*` variables (in particular `CUSTOM_ENV_CI_JOB_ID`), but neither Runner 17.11.3 nor 18.9.0 seem to populate those variables in the way the executor expects.
- The binary also refuses to run even basic commands like `config` unless these variables are present, including when called directly from the shell (outside of GitLab Runner).

This leads to two possible workarounds on my side:

- Introduce a wrapper script that maps standard `CI_*` environment variables to `CUSTOM_ENV_*` (e.g. `CUSTOM_ENV_CI_JOB_ID="$CI_JOB_ID"`), and point the custom executor to this wrapper instead of the binary directly.
- Try to rely on `--default-image` as hinted in the error message, but the current documentation does not clearly explain how this is supposed to work in the absence of `CUSTOM_ENV_*`.

My questions:

1. Is this behavior expected for `gitlab-tart-executor` 1.28.0 (always requiring `CUSTOM_ENV_CI_JOB_ID` for `config`/`prepare`), even when running from the CLI?
2. What is the exact expected contract between GitLab Runner and `gitlab-tart-executor` regarding environment variables (`CUSTOM_ENV_*` vs `CI_*`), especially for GitLab Runner 18.x?
3. Is there a documented way to use the executor without a wrapper (for example by correctly using `--default-image` or another flag) when `CUSTOM_ENV_*` is not present?
4. Is there a recommended `gitlab-tart-executor` version for GitLab Runner 18.x, or a specific Runner version officially supported for 1.28.0?

I’m happy to provide:

- Exact versions of GitLab Runner, Tart, macOS, and the full runner config,
- Additional logs (`--debug` on the runner, and full `gitlab-tart-executor` output for `prepare`/`run`),

if that helps reproducing or clarifying the expected behavior.

# EDIT : for refs

According to the official GitLab Runner Custom executor documentation, the driver is supposed to receive all CI/CD and predefined environment variables prefixed with ‎`CUSTOM_ENV_` rather than the raw ‎`CI_*` names.

The docs explicitly state that:

“Each executed step has access to specific environment variables that provide information about the running job. All stages have the following environment variables available to them:

– Standard CI/CD environment variables, including predefined variables.

– All environment variables provided by the Custom executor Runner host system.

– All services and their available settings. Exposed in JSON format as ‎`CUSTOM_ENV_CI_JOB_SERVICES`.

Both CI/CD environment variables and predefined variables are prefixed with ‎`CUSTOM_ENV_` to prevent conflicts with system environment variables. For example, ‎`CI_BUILDS_DIR` is available as ‎`CUSTOM_ENV_CI_BUILDS_DIR`.” [​⁠](https://docs.gitlab.com/runner/executors/custom/)

In other words, for the latest GitLab Runner and Custom executor implementation, the expected contract is that the driver reads metadata like ‎`CI_JOB_ID`, ‎`CI_PROJECT_DIR`, etc. via their ‎`CUSTOM_ENV_*` counterparts (for example ‎`CUSTOM_ENV_CI_JOB_ID`, ‎`CUSTOM_ENV_CI_PROJECT_DIR`), which are injected by GitLab into the driver’s environment for all stages (‎`config_exec`, ‎`prepare_exec`, ‎`run_exec`, ‎`cleanup_exec`).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the `config` and `prepare` entry points with and without `CUSTOM_ENV_CI_JOB_ID`, then compare the observed behavior with GitLab Runner's Custom executor environment contract. The payload names no source file or test; done means establishing whether the requirement is intentional and documenting or implementing a supported path for the reported Runner and Tart versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
gitlab, go, macos
Domain
ci-cd, devops, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.