hashicorp / hashicorp/nomad

document deployment behavior for lost allocs of previous job version

Open
#6,723 2 comments 0 reactions 0 assignees View on GitHub
theme/deployments theme/docs
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

In #6407 a lot of the work to investigate was building an understanding of the intended behavior of the scheduler during deployments. One of these behaviors is undocumented: when a deployment is running, a failed alloc for a previous job version is replaced with an alloc for the deployment's _new_ version, and not the previous version. The internal design doc does cover this behavior, but it never made it into a guide or doc:

> If there are migrating or lost allocations, the scheduler will replace them immediately, not counting against the max_parallelism, with the new job version. This is done for two reasons. It simplifies the scheduler and makes it so the scheduler only schedules the newest job which is easy for users to understand.
>
> Currently the scheduler will treat updates as evict and replace, unconditional on a replacement being feasible. This will have to be updated such that old versions are only stopped when a replacement is possible. This avoids an infeasible placement from rolling through and stopping all of the old job at a rate of max_parallelism / evaluation.

---

To reproduce the behavior, run Consul and Nomad in dev mode.

```sh
# start a job
nomad job run ./test.hcl

# wait for the deployment to succeed
nomad deployment status $id

# bump the env.version and redeploy
nomad job run ./test.hcl

# wait for all 3 allocs to be marked healthy, so that the
# deployment is pending manual approval. note that we
# have 6 running allocs
nomad job status test

# kill one of the old containers
docker kill $(docker ps | awk '/nginx/{print $1}' | tail -1)

# wait for Nomad to replace the container... it'll be version 1 and not version 0.
nomad job status test
```

jobspec

```
job "test" {
datacenters = ["dc1"]

group "webservers" {
count = 3

task "nginx" {
driver = "docker"

config {
image = "nginx:latest"

port_map = {
http = 80
}
}

env {
version = "0"
}

service {
name = "nginx"
port = "http"

check {
type = "http"
port = "http"
path = "/"
interval = "5s"
timeout = "3s"

check_restart {
limit = 1
grace = "5s"
ignore_warnings = false
}
}
}

resources {
memory = 64

network {
mbits = 10
port "http"{}
}
}
}

restart {
attempts = 0
delay = "10s"
}

update {
max_parallel = 3
health_check = "checks"
min_healthy_time = "5s"
healthy_deadline = "30s"
progress_deadline = "2m"
auto_revert = false
auto_promote = false
canary = 3
}
}
}

```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the deployment behavior described in issue #6407 and reproduce it using Consul and Nomad in dev mode, the provided test.hcl jobspec, and the listed nomad job and deployment commands. Document that a failed allocation from the previous job version is replaced by the deployment's new version, and include the reproduction steps and expected result in a user-facing guide.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, shell
Domain
devops, documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.