Self-hosted runner environment variables are not available during docker container setup
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 1.4k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 24
Description
Describe the bug
If you define environment variables in a .env file for a self-hosted runner they can be accessed as $VARIABLE during job steps, but if you want to access the variable when setting up the job in a docker container it is not resolved and instead remains as the string $VARIABLE rather than the value of VARIABLE. See the following example.
For a self hosted runner .env file such as:
DOCKER_GPUS=device=0
And a job defined as
jobs:
run_test:
name: Run
runs-on: self-hosted
container:
image: ubuntu
options: --gpus $DOCKER_GPUS
steps:
- name: Print Hello World
run: echo "Hello World!"
Running this will fail because $DOCKER_GPUS will be passed as a string to the docker --gpus flag, see excerpt from the job log:
Why is this a problem? We have servers with multiple GPUs and one self-hosted runner per GPU. Since self-hosted runners don't support docker-in-docker they all run on bare metal, while all the jobs are launched in docker containers using a setup like the above. We don't want to give all jobs access to all gpus (--gpus all) for multiple reasons, so therefore we defined the .env variables for each runner to be device=0, device=1, etc... to only expose a single GPU to jobs running on each runner. However, due to the problem above this doesn't work.
Further, this issue also means that you can't pass the runners' env variables into the docker container if you need to, like --env "MAX_CORES=$MAX_CORES", and similarly doing as follows also doesn't work:
container:
image: ubuntu
env:
cores: $MAX_CORES
Of course running a job without a container gives access to the env variable as expected:
jobs:
run_test:
name: Run
runs-on: self-hosted
steps:
- name: Print GPUs
run: echo "$DOCKER_GPUS"
To Reproduce
Steps to reproduce the behavior:
- Use a machine with a GPU installed
- Create a self hosted runner and add
DOCKER_GPUS=device=0to the.envfile - Create a workflow containing my first example above
- Run the workflow and see that it fails to read the environment variable
Alternatively if you don't have a GPU available you can also try to pass in an environment variable instead using --env "A=$B" and print its contents in the step, this will also fail.
Expected behavior
The variables in the .env file of the self-hosted runner should be accessible when the docker container is launched. If so, my example workflow above would have completed successfully.
Runner Version and Platform
Runner version: 2.313.0
Hosted on Ubuntu 22.04 machines.
What's not working?
See above.
Job Log Output
Log output on failure (same image as above):
Runner and Worker's Diagnostic Logs
Not relevant, can complement if necessary.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source files or tests are named in the issue. Start by reproducing the self-hosted Ubuntu 22.04 workflow with a runner .env value and a container options or env reference, then trace the runner's Docker container setup path. Done means the variable is expanded to its .env value before Docker is launched, while the existing job-step behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, docker
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100