hashicorp / hashicorp/nomad

stop unhealthy allocs first when reducing count

Open
#8,586 6 comments 0 reactions 0 assignees View on GitHub
stage/needs-discussion theme/deployments type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v0.11.1 (b43457070037800fcc8442c8ff095ff4005dab33)

### Operating system and Environment details
RHEL 7 x64

### Issue
If a running job with count = 2 specified on a group has one healthy alloc, and one unhealthy alloc, changing count to 1 and resubmitting the job killed the one healthy alloc and left the unhealthy alloc. In my instance the healthy alloc was old while the unhealthy alloc was fairly new, I'm not sure if this was coincidence or if the logic prefers to kill older allocs.

In my mind, it would make sense when reducing count to prefer to kill unhealthy allocs first.

### Reproduction steps
Start a job with count = 2 on a group
Contrive to make one of the allocs be marked as unhealthy - in my case, it is expected that the allocs take a while to startup (tens of minutes) and an old alloc was lost due to the failure of a nomad agent resulting in a replacement alloc being scheduled, but it had not yet successfully started.
Reduce the count on the group to 1
Observe that the healthy alloc may have been the one killed instead of the unhealthy alloc

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Start by reproducing the allocation-count reduction with one healthy and one unhealthy allocation, then trace the logic that selects allocations when reducing a group's count. Done means reducing the count preserves the healthy allocation and removes the unhealthy one, with regression coverage for this scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.