cirruslabs / cirruslabs/gitlab-tart-executor
CUSTOM_ENV_* requirements breaking custom executor with GitLab Runner
- Dominant language
- Go
- Stars
- 102
- Forks
- 14
- Avg merge
- 21h 24m
- Merged PRs (30d)
- 1
Description
### Context
I’m having trouble using `gitlab-tart-executor` as a custom executor with GitLab Runner on macOS (local development machine). After several attempts with different Runner and Tart versions, the blocker clearly appears inside `gitlab-tart-executor` itself, at the level of the `CUSTOM_ENV_*` environment handling.
The goal is to use CirrusLabs macOS images (Sequoia/Tahoe) in GitLab CI via `gitlab-tart-executor` on a macOS Apple Silicon machine (M series).
### Observed behavior
1. The executor works correctly for the `--version` command:
```bash
/opt/homebrew/bin/gitlab-tart-executor --version
executor version 1.28.0-6e163f8
```
2. However, even a simple `config` fails immediately when `CUSTOM_ENV_CI_JOB_ID` is not present in the environment, including outside of GitLab Runner:
```bash
/opt/homebrew/bin/gitlab-tart-executor config
GitLab environment error: CUSTOM_ENV_CI_JOB_ID is missing
```
3. In GitLab CI, with the Runner configured in `executor = "custom"` and calling `gitlab-tart-executor` directly for `config_exec` / `prepare_exec` / `run_exec` / `cleanup_exec`, the job always fails with:
```text
Preparing the "custom" executor
Using Custom executor with driver tart 1.28.0-6e163f8...
"prepare" stage failed: CUSTOM_ENV_CI_JOB_ID is missing and no --default-image was set
WARNING: Cleanup script failed: exit status 2
ERROR: Job failed: exit status 1
```
4. In the runner logs, you can see the job being picked up and then going straight to cleanup, without any visible Tart VM creation:
```text
Checking for jobs... received ...
Added job to processing list ...
WARNING: Failed to stop VM: tart command returned non-zero exit code: "the specified VM \"gitlab-\" does not exist"
WARNING: Failed to delete VM: VM errored: failed to delete VM gitlab-: tart command returned non-zero exit code: "the specified VM \"gitlab-\" does not exist"
WARNING: VM errored: failed to delete VM gitlab-: tart command returned non-zero exit code: "the specified VM \"gitlab-\" does not exist"
WARNING: Cleanup script failed: exit status 2
WARNING: Job failed: exit status 1
```
In other words, `gitlab-tart-executor` refuses to run `config`/`prepare` as long as `CUSTOM_ENV_CI_JOB_ID` (and potentially other `CUSTOM_ENV_*` variables) are not defined. The error message also mentions that no `--default-image` was set.
### GitLab Runner configuration (excerpt)
Here is an example of my current runner configuration:
```toml
concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "MacBook-M1Max-Tart"
url = "https://gitlab.com"
id = 52312176
token = "REDACTED"
executor = "custom"
environment = ["TART_EXECUTOR_SOFTNET=true", "TART_EXECUTOR_ALWAYS_PULL=false"]
[runners.feature_flags]
FF_RESOLVE_FULL_TLS_CHAIN = false
[runners.cache]
MaxUploadedArchiveSize = 0
[runners.cache.s3]
[runners.cache.gcs]
[runners.cache.azure]
[runners.custom]
config_exec = "/opt/homebrew/bin/gitlab-tart-executor"
config_args = ["config"]
prepare_exec = "sudo"
prepare_args = [
"/opt/homebrew/bin/gitlab-tart-executor", "prepare",
"--user", "juliengoudet",
"--concurrency", "1",
"--cpu", "auto",
"--memory", "auto"
]
run_exec = "sudo"
run_args = [
"/opt/homebrew/bin/gitlab-tart-executor", "run",
"--user", "juliengoudet"
]
cleanup_exec = "/opt/homebrew/bin/gitlab-tart-executor"
cleanup_args = ["cleanup"]
```
The Tart CLI itself (`tart`) works fine when used directly to launch CirrusLabs macOS images on this machine.
### What I already tried
- GitLab Runner 18.8.0 / 18.9.0: same error in CI
`"prepare" stage failed: CUSTOM_ENV_CI_JOB_ID is missing and no --default-image was set`.
- Downgraded GitLab Runner to 17.11.3: same behavior, i.e. `gitlab-tart-executor config` and `prepare` refuse to start without `CUSTOM_ENV_CI_JOB_ID` being defined.
- Verified `sudoers` and permissions:
`sudo /opt/homebrew/bin/gitlab-tart-executor --version` works without a password, but `config` still fails with the same `CUSTOM_ENV_*` error.
- Manual tests with `CUSTOM_ENV_CI_JOB_ID` defined in the environment before calling `gitlab-tart-executor config`: in that case, `config` starts working, which confirms that the hard requirement on `CUSTOM_ENV_*` is enforced inside the executor.
### Problem / questions
From what I can see:
- `gitlab-tart-executor` 1.28.0-6e163f8 is designed to be invoked in an environment where GitLab provides the `CUSTOM_ENV_*` variables (in particular `CUSTOM_ENV_CI_JOB_ID`), but neither Runner 17.11.3 nor 18.9.0 seem to populate those variables in the way the executor expects.
- The binary also refuses to run even basic commands like `config` unless these variables are present, including when called directly from the shell (outside of GitLab Runner).
This leads to two possible workarounds on my side:
- Introduce a wrapper script that maps standard `CI_*` environment variables to `CUSTOM_ENV_*` (e.g. `CUSTOM_ENV_CI_JOB_ID="$CI_JOB_ID"`), and point the custom executor to this wrapper instead of the binary directly.
- Try to rely on `--default-image` as hinted in the error message, but the current documentation does not clearly explain how this is supposed to work in the absence of `CUSTOM_ENV_*`.
My questions:
1. Is this behavior expected for `gitlab-tart-executor` 1.28.0 (always requiring `CUSTOM_ENV_CI_JOB_ID` for `config`/`prepare`), even when running from the CLI?
2. What is the exact expected contract between GitLab Runner and `gitlab-tart-executor` regarding environment variables (`CUSTOM_ENV_*` vs `CI_*`), especially for GitLab Runner 18.x?
3. Is there a documented way to use the executor without a wrapper (for example by correctly using `--default-image` or another flag) when `CUSTOM_ENV_*` is not present?
4. Is there a recommended `gitlab-tart-executor` version for GitLab Runner 18.x, or a specific Runner version officially supported for 1.28.0?
I’m happy to provide:
- Exact versions of GitLab Runner, Tart, macOS, and the full runner config,
- Additional logs (`--debug` on the runner, and full `gitlab-tart-executor` output for `prepare`/`run`),
if that helps reproducing or clarifying the expected behavior.
# EDIT : for refs
According to the official GitLab Runner Custom executor documentation, the driver is supposed to receive all CI/CD and predefined environment variables prefixed with `CUSTOM_ENV_` rather than the raw `CI_*` names.
The docs explicitly state that:
“Each executed step has access to specific environment variables that provide information about the running job. All stages have the following environment variables available to them:
– Standard CI/CD environment variables, including predefined variables.
– All environment variables provided by the Custom executor Runner host system.
– All services and their available settings. Exposed in JSON format as `CUSTOM_ENV_CI_JOB_SERVICES`.
Both CI/CD environment variables and predefined variables are prefixed with `CUSTOM_ENV_` to prevent conflicts with system environment variables. For example, `CI_BUILDS_DIR` is available as `CUSTOM_ENV_CI_BUILDS_DIR`.” [](https://docs.gitlab.com/runner/executors/custom/)
In other words, for the latest GitLab Runner and Custom executor implementation, the expected contract is that the driver reads metadata like `CI_JOB_ID`, `CI_PROJECT_DIR`, etc. via their `CUSTOM_ENV_*` counterparts (for example `CUSTOM_ENV_CI_JOB_ID`, `CUSTOM_ENV_CI_PROJECT_DIR`), which are injected by GitLab into the driver’s environment for all stages (`config_exec`, `prepare_exec`, `run_exec`, `cleanup_exec`).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the `config` and `prepare` entry points with and without `CUSTOM_ENV_CI_JOB_ID`, then compare the observed behavior with GitLab Runner's Custom executor environment contract. The payload names no source file or test; done means establishing whether the requirement is intentional and documenting or implementing a supported path for the reported Runner and Tart versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gitlab, go, macos
- Domain
- ci-cd, devops, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100