actions / actions/runner

Anonymous volumes not cleaned up after containers are removed

Open
#1,885 8 comments 13 reactions 1 assignee View on GitHub

@ruvceskistefan is already working on this.

Since May 25, 2022.

bug Runner Bug
Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

When using a service container with a VOLUME declaration in its dockerfile
(both the postgresql and redis containers used in the examples at
https://docs.github.com/en/actions/using-containerized-services do this), an
anonymous volume is automatically created together with the container.

At the end of the workflow, the container is stopped and removed, but the
anonymous volumes it created stay around.

In our case, we use a postgres service as a throwaway database for some tests,
which results in many smallish anonymous volumes accumulating on the systems
that host our GH runners.

At the moment we work around this with manual cleanup or cronjobs, but ideally
these containers would be fully removed at the end of workflows.

I tried passing --rm to jobs.<job_id>.services.<service_id>.options, but
that ends up as an argument to docker create. The runner creates, starts,
stops and removes containers with separate commands, so passing --rm to
docker create has no effect on docker remove.

Would it be an option to change DockerRemove to use the -v option?

https://github.com/actions/runner/blob/628f462ab709492bf03b149468ef18415f9bc1bb/src/Runner.Worker/Container/DockerCommandManager.cs#L266-L269

From the docker rm docs:

--volumes , -v Remove anonymous volumes associated with the container

Any idea for a workaround would also be appreciated, I don't see any other
way to have these anonymous volumes removed.

Thank you!

To Reproduce

Run a workflow with a container which uses VOLUME, for example (from https://docs.github.com/en/actions/using-containerized-services/creating-postgresql-service-containers):

    container: node:10.18-jessie
    # Service containers to run with `container-job`
    services:
      # Label used to access the service container
      postgres:
        # Docker Hub image
        image: postgres
        # Provide the password for postgres
        env:
          POSTGRES_PASSWORD: postgres
        # Set health checks to wait until postgres has started
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

After the workflow has finished, check with docker system df -v on the machine that ran the job, the anonymous volume created by the service container is still around:

VOLUME NAME                                                        LINKS     SIZE
b9837a9103b4f4a863a93dd952e7b8777c10274fc428cac4760ae79f13ac5826   0         43.2MB

Expected behavior

Anonymous volumes created automatically by the service container should be cleaned up.

Runner Version and Platform

2.291.1, Linux

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.