Disconnected allocations are not replaced once resources are made available
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
```
Nomad v1.5.2
BuildDate 2023-03-21T22:54:38Z
Revision 9a2fdb5f53dce81edf2802f0b64962e07596fd03
```
### Operating system and Environment details
MacOS
### Issue
During a canary deployment that is pending promotion, if a client that has allocations with `max_client_disconnect` set is disconnected, and there are no available capacity to replace these allocations, the allocations are never replaced even when capacity is made available later on.
### Reproduction steps
Start 1 client and 1 server.
```hcl
name = "server1"
data_dir = "/tmp/nomad/server1"
server {
enabled = 1
bootstrap_expect = 1
}
client {
enabled = false
}
```
```hcl
name = "client1"
data_dir = "/tmp/nomad/client1"
ports {
http = 4656
rpc = 4657
serf = 4658
}
server {
enabled = false
}
client {
enabled = true
server_join {
retry_join = ["127.0.0.1"]
}
}
```
Run sample job with `max_client_disconnect` and an `update` block.
```hcl
job "example" {
group "cache" {
max_client_disconnect = "12h"
network {
port "db" {
to = 6379
}
}
update {
canary = 1
auto_promote = false
}
task "redis" {
driver = "docker"
config {
image = "redis:7"
ports = ["db"]
}
}
}
}
```
Update job (for example, change `image` to `redis:6`). Once deployment is healthy and awaiting promotion, stop client. Allocations will eventually be marked as `unknown`.
Start a second client.
```hcl
name = "client2"
data_dir = "/tmp/nomad/client2"
ports {
http = 4666
rpc = 4667
serf = 4668
}
server {
enabled = false
}
client {
enabled = true
server_join {
retry_join = ["127.0.0.1"]
}
}
```
#### Expected Result
Allocations are placed in the new client.
#### Actual Result
No allocations are placed until the deployment is either promoted or failed.
### Nomad Server logs (if appropriate)
The lack of log lines is the interesting point. The [`createNodeEvals`](https://github.com/hashicorp/nomad/blob/v1.5.2/nomad/node_endpoint.go#L1593-L1705) needs to handle this case so a `node-update` eval is created for the job being deployed.
```
2023-03-24T18:30:23.406-0400 [DEBUG] worker.service_sched: reconciled current state with desired state: eval_id=49ed55ce-87b7-8b2e-6953-6b6e49f9f8e1 job_id=example namespace=default worker_id=6f88259f-4d38-b901-ebfb-a50522bf87d1
results=
| Total changes: (place 2) (destructive 0) (inplace 0) (stop 0) (disconnect 2) (reconnect 0)
| Desired Changes for "cache": (place 2) (inplace 0) (destructive 0) (stop 0) (migrate 0) (ignore 0) (canary 0)
2023-03-24T18:30:23.414-0400 [DEBUG] http: request complete: method=GET path=/v1/node/61d9836c-990a-abca-6686-f0c1470826a4 duration="171.417µs"
2023-03-24T18:30:23.417-0400 [DEBUG] worker: created evaluation: worker_id=6f88259f-4d38-b901-ebfb-a50522bf87d1 eval=""
2023-03-24T18:30:23.417-0400 [DEBUG] worker.service_sched: found reschedulable allocs, followup eval created: eval_id=49ed55ce-87b7-8b2e-6953-6b6e49f9f8e1 job_id=example namespace=default worker_id=6f88259f-4d38-b901-ebfb-a50522bf87d1 followup_eval_id=bed88082-27f5-1bda-1fed-18a013abf3cf
2023-03-24T18:30:23.417-0400 [TRACE] nomad: evaluating plan: plan="(eval 49ed55ce, job example, NodeAllocations: (node[61d9836c] (43732d8b example.cache[0] run) (cc86ecc3 example.cache[0] run)))"
2023-03-24T18:30:23.433-0400 [DEBUG] worker: submitted plan for evaluation: worker_id=6f88259f-4d38-b901-ebfb-a50522bf87d1 eval_id=49ed55ce-87b7-8b2e-6953-6b6e49f9f8e1
2023-03-24T18:30:23.433-0400 [DEBUG] worker.service_sched: setting eval status: eval_id=49ed55ce-87b7-8b2e-6953-6b6e49f9f8e1 job_id=example namespace=default worker_id=6f88259f-4d38-b901-ebfb-a50522bf87d1 status=complete
2023-03-24T18:30:23.433-0400 [DEBUG] http: request complete: method=GET path=/v1/job/example/summary?index=25 duration=12.403996292s
2023-03-24T18:30:23.433-0400 [DEBUG] http: request complete: method=GET path=/v1/job/example/allocations?index=27 duration=14.399747042s
2023-03-24T18:30:23.433-0400 [DEBUG] http: request complete: method=GET path="/v1/jobs?meta=true&index=25" duration=16.118075708s
2023-03-24T18:30:23.433-0400 [DEBUG] http: request complete: method=GET path=/v1/node/61d9836c-990a-abca-6686-f0c1470826a4/allocations?index=26 duration=5.680428792s
2023-03-24T18:30:23.442-0400 [DEBUG] worker: updated evaluation: worker_id=6f88259f-4d38-b901-ebfb-a50522bf87d1 eval=""
2023-03-24T18:30:23.442-0400 [DEBUG] worker: ack evaluation: worker_id=6f88259f-4d38-b901-ebfb-a50522bf87d1 eval_id=49ed55ce-87b7-8b2e-6953-6b6e49f9f8e1 type=service namespace=default job_id=example node_id=61d9836c-990a-abca-6686-f0c1470826a4 triggered_by=node-update
2023-03-24T18:30:23.442-0400 [TRACE] worker: changed workload status: worker_id=6f88259f-4d38-b901-ebfb-a50522bf87d1 from=Scheduling to=WaitingToDequeue
---- New node registers unblocking the clients UI view ---
2023-03-24T18:30:37.524-0400 [DEBUG] http: request complete: method=GET path=/v1/nodes?index=29 duration=14.115290916s
2023-03-24T18:30:37.560-0400 [DEBUG] http: request complete: method=GET path=/v1/node/e82f0ff4-50cd-d8ad-be2f-5924041950d4/allocations duration="159.958µs"
2023-03-24T18:30:37.564-0400 [DEBUG] http: request complete: method=GET path=/v1/node/e82f0ff4-50cd-d8ad-be2f-5924041950d4 duration="259.958µs"
2023-03-24T18:30:37.565-0400 [DEBUG] http: request complete: method=GET path=/v1/node/61d9836c-990a-abca-6686-f0c1470826a4 duration="148.292µs"
2023-03-24T18:30:37.637-0400 [DEBUG] http: request complete: method=GET path=/v1/nodes?index=35 duration=88.838959ms
2023-03-24T18:30:37.647-0400 [DEBUG] http: request complete: method=GET path=/v1/node/e82f0ff4-50cd-d8ad-be2f-5924041950d4 duration="173.25µs"
2023-03-24T18:30:37.647-0400 [DEBUG] http: request complete: method=GET path=/v1/node/61d9836c-990a-abca-6686-f0c1470826a4 duration="151.291µs"
2023-03-24T18:30:43.092-0400 [DEBUG] http: request complete: method=GET path=/v1/nodes?index=36 duration=3.532947958s
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with createNodeEvals in nomad/node_endpoint.go, then run the provided two-client reproduction with a canary deployment and max_client_disconnect. Trace why no node-update evaluation is created when the replacement client registers. Done means disconnected allocations are placed on the new client before the deployment is promoted or failed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100