hashicorp / hashicorp/nomad

Jobs with Canaries may not be rescheduled

Open
#8,671 2 comments 3 reactions 0 assignees View on GitHub
hcc/cst stage/accepted theme/core theme/deployments theme/scheduling type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

In somewhat an edge case, failed or lost allocations for jobs with pending promotion deployments may never be rescheduled, if the cluster is oversubscribed and they cannot be immediately migrated. The bug seems that blocking eval processing doesn't handle case where jobs have non-promoted deployment well.

I made a sample "unit" test in https://github.com/hashicorp/nomad/compare/b-canary-eligible-reschedule?expand=1 , and it's failing in https://app.circleci.com/pipelines/github/hashicorp/nomad/11168/workflows/038a0ba7-3da4-44ff-9f11-22d12412b031/jobs/91890 .

Simply forcing an eval or resubmitting the job doesn't help. One must submit a job with an update to place the job allocs properly.

### Steps to reproduce in a cluster
1. Spin up a cluster with a single client
2. Invoke a sample service job, like the example job below
3. Change the image to `redis:5`, and resubmit
4. Notice 3 running allocs with version 0, and 1 canary alloc with version 2
5. Kill the client
6. Wait until it's marked lost
7. Add a new client agent
8. Run `nomad job eval example`

We'd expect the allocations to be migrated to new client. But nomad doesn't run any.

### Job file

```hcl
job "example" {
datacenters = ["dc1"]

update {
max_parallel = 1
canary = 1
auto_promote = false
auto_revert = false
}

group "group1" {
count = 3
task "redis" {
driver = "docker"
config { image = "redis:4" }
}
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the sample unit test in the linked b-canary-eligible-reschedule comparison and reproduce its failure using the CircleCI job or the cluster steps in this issue. Trace blocking evaluation processing for a job with a non-promoted deployment and verify that lost allocations are migrated after a replacement client joins and an evaluation runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.