Nomad restart caused docker image clean up to no longer run
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
0.12.5
### Operating system and Environment details
PhotonOS3
### Issue
A little while ago we reported issue https://github.com/hashicorp/nomad/issues/9045 which was subsequently fixed. Things have been working great until we needed to restart our Nomad agents - After which we noticed that images were being left behind and not cleaned up. We have been reliably able to reproduce the issue with the steps below. This seems to also be related to how Nomad handles state after a restart (see https://github.com/hashicorp/nomad/issues/9491)
### Reproduction steps
1. Set the Nomad Image GC to 1 min (allows reasonable time based testing)
2. Run a job that schedules a container.
3. ```docker image ls``` should list the container.
4. Stop the job
5. ```docker image ls``` should still list the container
6. Wait 1 min and verify that container is gone as expected.
Now repeat the test BUT after step 3, restart the nomad agent, then continue the test. The image will never be cleaned up.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Docker image cleanup behavior with Nomad 0.12.5 and a one-minute Image GC interval, first normally and then after restarting the Nomad agent. Trace how image GC handles state across the restart. Done means stopped-job images are removed after the configured interval in both cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100