Jobs with Canaries may not be rescheduled
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
In somewhat an edge case, failed or lost allocations for jobs with pending promotion deployments may never be rescheduled, if the cluster is oversubscribed and they cannot be immediately migrated. The bug seems that blocking eval processing doesn't handle case where jobs have non-promoted deployment well.
I made a sample "unit" test in https://github.com/hashicorp/nomad/compare/b-canary-eligible-reschedule?expand=1 , and it's failing in https://app.circleci.com/pipelines/github/hashicorp/nomad/11168/workflows/038a0ba7-3da4-44ff-9f11-22d12412b031/jobs/91890 .
Simply forcing an eval or resubmitting the job doesn't help. One must submit a job with an update to place the job allocs properly.
### Steps to reproduce in a cluster
1. Spin up a cluster with a single client
2. Invoke a sample service job, like the example job below
3. Change the image to `redis:5`, and resubmit
4. Notice 3 running allocs with version 0, and 1 canary alloc with version 2
5. Kill the client
6. Wait until it's marked lost
7. Add a new client agent
8. Run `nomad job eval example`
We'd expect the allocations to be migrated to new client. But nomad doesn't run any.
### Job file
```hcl
job "example" {
datacenters = ["dc1"]
update {
max_parallel = 1
canary = 1
auto_promote = false
auto_revert = false
}
group "group1" {
count = 3
task "redis" {
driver = "docker"
config { image = "redis:4" }
}
}
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the sample unit test in the linked b-canary-eligible-reschedule comparison and reproduce its failure using the CircleCI job or the cluster steps in this issue. Trace blocking evaluation processing for a job with a non-promoted deployment and verify that lost allocations are migrated after a replacement client joins and an evaluation runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100