stop unhealthy allocs first when reducing count
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
Nomad v0.11.1 (b43457070037800fcc8442c8ff095ff4005dab33)
### Operating system and Environment details
RHEL 7 x64
### Issue
If a running job with count = 2 specified on a group has one healthy alloc, and one unhealthy alloc, changing count to 1 and resubmitting the job killed the one healthy alloc and left the unhealthy alloc. In my instance the healthy alloc was old while the unhealthy alloc was fairly new, I'm not sure if this was coincidence or if the logic prefers to kill older allocs.
In my mind, it would make sense when reducing count to prefer to kill unhealthy allocs first.
### Reproduction steps
Start a job with count = 2 on a group
Contrive to make one of the allocs be marked as unhealthy - in my case, it is expected that the allocs take a while to startup (tens of minutes) and an old alloc was lost due to the failure of a nomad agent resulting in a replacement alloc being scheduled, but it had not yet successfully started.
Reduce the count on the group to 1
Observe that the healthy alloc may have been the one killed instead of the unhealthy alloc
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named. Start by reproducing the allocation-count reduction with one healthy and one unhealthy allocation, then trace the logic that selects allocations when reducing a group's count. Done means reducing the count preserves the healthy allocation and removes the unhealthy one, with regression coverage for this scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100