hashicorp / hashicorp/nomad

Nomad restart caused docker image clean up to no longer run

Open
#9,549 2 comments 0 reactions 0 assignees View on GitHub
stage/accepted stage/needs-discussion theme/driver/docker
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
0.12.5

### Operating system and Environment details
PhotonOS3

### Issue
A little while ago we reported issue https://github.com/hashicorp/nomad/issues/9045 which was subsequently fixed. Things have been working great until we needed to restart our Nomad agents - After which we noticed that images were being left behind and not cleaned up. We have been reliably able to reproduce the issue with the steps below. This seems to also be related to how Nomad handles state after a restart (see https://github.com/hashicorp/nomad/issues/9491)

### Reproduction steps
1. Set the Nomad Image GC to 1 min (allows reasonable time based testing)
2. Run a job that schedules a container.
3. ```docker image ls``` should list the container.
4. Stop the job
5. ```docker image ls``` should still list the container
6. Wait 1 min and verify that container is gone as expected.

Now repeat the test BUT after step 3, restart the nomad agent, then continue the test. The image will never be cleaned up.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Docker image cleanup behavior with Nomad 0.12.5 and a one-minute Image GC interval, first normally and then after restarting the Nomad agent. Trace how image GC handles state across the restart. Done means stopped-job images are removed after the configured interval in both cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.