improve user control when scaling down task groups
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Issue
Consider a nomad job containing all tasks for a capability including a group corresponding to a worker pool. This worker pool processes long duration tasks from a queue. An external process can monitor application state (like queue depth) and resubmit the job with a differing count to scale the pool up or down. This works well, for the scale up, but causes trouble scaling down with long duration tasks.
Specifically, there is no control over _which_ task groups get eliminated in a scale down event. Ideally you'd have some influence so that you could take down idle workers rather than interrupt ones already well into a long duration task.
### Some things I've tried thus far
Using nomad 0.11.1 running in dev mode...
1. Stopping an allocation and then immediately scaling down hoping the scheduler would remove that allocation. It doesn't seem to, instead it seems to prefer taking the most recent allocation regardless of state (pending, failed, etc.)
2. Banking on the "removal of newest" behavior, I tried doing a restart of the allocation I wanted to remove, hoping to make the most recent, but this also doesn't work. The tracking doesn't seem to consider the reallocation as newer.
3. I tried changing my paradigm a bit. I changed workers to self-destruct when idle for a sufficient amount of time and in nomad job I disabled restart/reschedule behaviors. I thought I could then have an external controller monitor if too many went down and resubmit the nomad job with desired counts. Unfortunately I found nomad is happy to count the dead allocations against the desired worker count rather than deploy new allocations with seemingly no way to re-kick those failed allocations. Specifically they don't honor any updates to the job restart/reschedule configuration and explicitly allocation restart behaviors only apply to presently running allocations.
4. Took a good look at the new auto-scale capabilities, but it looks to have all the same limitations.
### Solution Ideas
I'd love to hear inputs on alternate ways to structure things to solve this issue, but I think regardless the ability to provide some input on removal decisions is a good thing. I don't really know the best way forward, but here are some starter options to consider:
1. Provide an explicit mechanism for marking or weighting allocations as better/worse candidate for removal to directly feed scheduler decisions.
2. Have the scheduler consider activity (like CPU) of allocations for take down. Generally I'd suspect allocations using less CPU are more likely to be idle and better candidates for elimination.
3. If elimination of newer allocations is a reliable behavior, make it so that re-allocations are indeed considered "newer".
Thoughts?
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named. Start by tracing Nomad's scheduler allocation-removal and restart/reschedule decisions, then compare the proposed removal weighting, activity-based selection, and reallocation-age ideas. Done would require a decided scope, documented behavior for scaling down task groups, and tests covering the selected allocation behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100