GoogleContainerTools / GoogleContainerTools/container-structure-test

Loaded OCI images are left orphaned in Docker daemon after test execution

Open
#560 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.5k
Forks
212
PR merge metrics
No merged PRs in 30d

Description

### Problem
When running container_structure_test targets with Bazel using the docker driver, the Bazel rule compiles a wrapper launcher script that loads the built OCI image layout into the local Docker daemon under a dynamic unique tag: `--default-image-tag "cst.oci.local/$DIGEST:$DIGEST"`. However, after test execution this image tag is not cleaned up from the Docker daemon.

This means that on developer machines and stateful runners (e.g. self-hosted GPU runners) this results in large amounts of disk space being used for leftover docker images.

```console
> docker images
i Info → U In Use
IMAGE ID DISK USAGE CONTENT SIZE
cst.oci.local/sha256-0f22b2b3ab798590eba542fd0b97c7eea9d75f4d22e265f8a6cba473e3d5d24f d460401f0e96 12.2GB 0B
cst.oci.local/sha256-19138e79d02af90ad4da61e5fc41b6fdb109956356dbf8be349ae2a095953041 bec76fe59b4b 12.2GB 0B
cst.oci.local/sha256-25ae02e0a51277d9a52238dbace09e82001804908971974cdcf921328f6dd31d 9d892964272a 12.2GB 0B
cst.oci.local/sha256-300734190c063608de30f95c297603b0c6be1527953c6eec9d067f1983ddd28b 8cc6a7358615 12.2GB 0B
cst.oci.local/sha256-483f8a0141c030e23cb1ea331b48271fd5c75913b2911fe4b360946bc278d496 abc7645425fb 12.2GB 0B
cst.oci.local/sha256-4e4d63413de21ce825d8ce1ac8eabd1766e7b577124501b60cdfffd44d723c3f 4c1bad1d8747 12.2GB 0B
...
cst.oci.local/sha256-d0e9b5def972bfa86bf1d38f07a8541f3b1d2ffabcc3851095c679e33898ae38 cf9f322bd3b7 11.3GB 0B
cst.oci.local/sha256-d5df6e7bb4968ac1ab923a6073ffe9fbb984b60ffc19756cdaf5f5f6497385da dcc785d69393 12.2GB 0B
cst.oci.local/sha256-dd00cdb24084dae5b00f81f04a40913917ff03d7943cd220b16e5e35fa92c167 2c6ca4cb8d2b 12.2GB 0B
cst.oci.local/sha256-ee066855ba10357ee64115f3b5df1c2fbcb1f6fef0bab1eb04e1bb382b7d75ce 1435fcd64b84 12.2GB 0B
cst.oci.local/sha256-f42d968b033ac86f3ee0233d3e1200bf7d6d49e318d67c307d76f731fe615c2e e729997dda2b 12.2GB 0B
cst.oci.local/sha256-fccd3bb94eafab0006c3f56dcf3626df3c4142c758ad9a6c060a4a29d0e9d54d ad5ffafb4631 12.2GB 0B
cst.oci.local/sha256-fd1526406fe8aac0bf3f8783b6b31c0348282be5d093a6ee941219adb6236aab f6fa3e41e9c6 12.2GB 0B
cst.oci.local/sha256-fdb5bdbffc96f4b276c9668a99881b1aad14c34d77927243a1d24fa7d52893d1 24f3206d5feb 12.2GB 0B
```

### Current workaround we are using

We're patching the `bazel/container_structure_test.bzl` with the following:
```patch
--- bazel/container_structure_test.bzl
+++ bazel/container_structure_test.bzl
@@ -37,9 +37,18 @@
# When the image points to a folder, we can read the index.json file inside
if [[ -d "$image" ]]; then
readonly DIGEST=$("$jq" -r '.manifests[0].digest | sub(":"; "-")' "$image/index.json")
- exec "$st" test --driver {driver} {fixed_args} --default-image-tag "cst.oci.local/$DIGEST:$DIGEST" $@
+ readonly TEST_TAG="cst.oci.local/$DIGEST:$DIGEST"
+
+ # Register a cleanup trap to remove the loaded image from Docker
+ cleanup() {{
+ docker rmi "$TEST_TAG" >/dev/null 2>&1 || true
+ }}
+ trap cleanup EXIT
+
+ "$st" test --driver {driver} {fixed_args} --default-image-tag "$TEST_TAG" "$@"
+ exit "$?"
else
- exec "$st" test --driver {driver} {fixed_args} $@
+ exec "$st" test --driver {driver} {fixed_args} "$@"
fi
"""
```
effectively ensuring the created docker image is removed after the test completes

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.