hashicorp / hashicorp/nomad

Enable Nomad with multi container image sources for high availability

Open
#17,839 5 comments 4 reactions 0 assignees View on GitHub
stage/accepted theme/driver/docker type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Proposal
Allow Nomad to have fallback docker image urls to pull from. Lately Gitlab have been somewhat down now and then. Although this is not linked with Nomad itself, it could be an opportunity for improvement. Changing the image config in the docker driver block to become a list instead of a single string so Nomad can try multiple sources from which to pull the container image. Pushing a container to multiple registries is easy to get done in a CI/CD pipeline and is far more convenient than re-evaluating your CI/CD or container registry service provider. (or simply not worth it just for a registry being down or slow occasionally)

Changing:
```
config {
image = "https://hub.docker.internal/redis:7"
}
```
To:
```
config {
image = ["https://hub.docker.internal/redis:7", https://registry.gitlab.com/redis:7", https://gcr.io/redis:7"]
}
```
Could also be:
```
config {
image = ["https://registry1.my.organisation/redis:7", https://registry2.my.organisation/redis:7", https://registry3.my.organisation/redis:7"]
}
```

In the event that the 1st url cannot pull the image, Nomad would automatically try the 2nd url and so on. Would allow Nomad to provide its users with high availability even when the "highly available" service provider is not.

A `retry_pull:` config could also be used in conjunction with this feature to define the number of times to retry pulling from a source before switching to the next.

### Use-cases
- Nomad provides an extra availability layer to protect the user from a crash on the 3rd party side even if it rarely happens.
- Nomad can load balance itself when pulling from a busy registry infrastructure.
- Is aligned with the multi cloud paradigm.

### Attempted Solutions
Could include pulling the image on the target host in the CI/CD pipeline so Nomad does not have to do it.
Could self host a registry but this would add weight on the infrastructure management.
Could consider re-evaluating another service provider but could be incredibly painful or even in some cases just not feasible (Probably the worst solution).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading Nomad's Docker driver configuration and image-pull path, focusing on how the current single image value is handled. Define how multiple registry URLs and optional retry behavior should work, then verify that failed pulls fall back in order without breaking existing single-image configurations.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.