hashicorp / hashicorp/nomad

improve user control when scaling down task groups

Open
#7,876 2 comments 0 reactions 0 assignees View on GitHub
stage/needs-discussion theme/autoscaling theme/core
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Issue

Consider a nomad job containing all tasks for a capability including a group corresponding to a worker pool. This worker pool processes long duration tasks from a queue. An external process can monitor application state (like queue depth) and resubmit the job with a differing count to scale the pool up or down. This works well, for the scale up, but causes trouble scaling down with long duration tasks.

Specifically, there is no control over _which_ task groups get eliminated in a scale down event. Ideally you'd have some influence so that you could take down idle workers rather than interrupt ones already well into a long duration task.

### Some things I've tried thus far

Using nomad 0.11.1 running in dev mode...

1. Stopping an allocation and then immediately scaling down hoping the scheduler would remove that allocation. It doesn't seem to, instead it seems to prefer taking the most recent allocation regardless of state (pending, failed, etc.)

2. Banking on the "removal of newest" behavior, I tried doing a restart of the allocation I wanted to remove, hoping to make the most recent, but this also doesn't work. The tracking doesn't seem to consider the reallocation as newer.

3. I tried changing my paradigm a bit. I changed workers to self-destruct when idle for a sufficient amount of time and in nomad job I disabled restart/reschedule behaviors. I thought I could then have an external controller monitor if too many went down and resubmit the nomad job with desired counts. Unfortunately I found nomad is happy to count the dead allocations against the desired worker count rather than deploy new allocations with seemingly no way to re-kick those failed allocations. Specifically they don't honor any updates to the job restart/reschedule configuration and explicitly allocation restart behaviors only apply to presently running allocations.

4. Took a good look at the new auto-scale capabilities, but it looks to have all the same limitations.

### Solution Ideas

I'd love to hear inputs on alternate ways to structure things to solve this issue, but I think regardless the ability to provide some input on removal decisions is a good thing. I don't really know the best way forward, but here are some starter options to consider:

1. Provide an explicit mechanism for marking or weighting allocations as better/worse candidate for removal to directly feed scheduler decisions.

2. Have the scheduler consider activity (like CPU) of allocations for take down. Generally I'd suspect allocations using less CPU are more likely to be idle and better candidates for elimination.

3. If elimination of newer allocations is a reliable behavior, make it so that re-allocations are indeed considered "newer".

Thoughts?

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Start by tracing Nomad's scheduler allocation-removal and restart/reschedule decisions, then compare the proposed removal weighting, activity-based selection, and reallocation-age ideas. Done would require a decided scope, documented behavior for scaling down task groups, and tests covering the selected allocation behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.