actions / actions/runner

Self-hosted runner environment variables are not available during docker container setup

Open
#3,159 4 comments 5 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Describe the bug
If you define environment variables in a .env file for a self-hosted runner they can be accessed as $VARIABLE during job steps, but if you want to access the variable when setting up the job in a docker container it is not resolved and instead remains as the string $VARIABLE rather than the value of VARIABLE. See the following example.

For a self hosted runner .env file such as:

DOCKER_GPUS=device=0

And a job defined as

jobs:
  run_test:
    name: Run
    runs-on: self-hosted
    container:
      image: ubuntu
      options: --gpus $DOCKER_GPUS
    steps:
      - name: Print Hello World
        run: echo "Hello World!"

Running this will fail because $DOCKER_GPUS will be passed as a string to the docker --gpus flag, see excerpt from the job log:
image

Why is this a problem? We have servers with multiple GPUs and one self-hosted runner per GPU. Since self-hosted runners don't support docker-in-docker they all run on bare metal, while all the jobs are launched in docker containers using a setup like the above. We don't want to give all jobs access to all gpus (--gpus all) for multiple reasons, so therefore we defined the .env variables for each runner to be device=0, device=1, etc... to only expose a single GPU to jobs running on each runner. However, due to the problem above this doesn't work.

Further, this issue also means that you can't pass the runners' env variables into the docker container if you need to, like --env "MAX_CORES=$MAX_CORES", and similarly doing as follows also doesn't work:

container:
  image: ubuntu
  env:
    cores: $MAX_CORES

Of course running a job without a container gives access to the env variable as expected:

jobs:
  run_test:
    name: Run
    runs-on: self-hosted
    steps:
      - name: Print GPUs
        run: echo "$DOCKER_GPUS"

To Reproduce
Steps to reproduce the behavior:

  1. Use a machine with a GPU installed
  2. Create a self hosted runner and add DOCKER_GPUS=device=0 to the .env file
  3. Create a workflow containing my first example above
  4. Run the workflow and see that it fails to read the environment variable

Alternatively if you don't have a GPU available you can also try to pass in an environment variable instead using --env "A=$B" and print its contents in the step, this will also fail.

Expected behavior
The variables in the .env file of the self-hosted runner should be accessible when the docker container is launched. If so, my example workflow above would have completed successfully.

Runner Version and Platform

Runner version: 2.313.0
Hosted on Ubuntu 22.04 machines.

What's not working?

See above.

Job Log Output

Log output on failure (same image as above):
image

Runner and Worker's Diagnostic Logs

Not relevant, can complement if necessary.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source files or tests are named in the issue. Start by reproducing the self-hosted Ubuntu 22.04 workflow with a runner .env value and a container options or env reference, then trace the runner's Docker container setup path. Done means the variable is expanded to its .env value before Docker is launched, while the existing job-step behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, docker
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.