hashicorp / hashicorp/nomad

Select a fixed number of alternative task groups within a service job

Open
#28,529 3 comments 0 reactions 0 assignees View on GitHub
hcc/jira stage/needs-discussion theme/scheduling type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Proposal

Allow a service job to declare alternative task groups and request a fixed number of them. Each selected group would retain its own constraints, tasks, replica count, and resources. Groups not listed in any selection would remain required.

For example, the following proposed syntax would select one of three implementations of the same service:

```hcl
group_selection "runtime" {
count = 1
groups = ["encoder", "orin", "thor"]
}
```

The three ordinary `group` blocks would declare their respective resources and node constraints. For a one-replica service, they might request 1000 MHz / 1000 MiB, 800 MHz / 800 MiB, and 500 MHz / 500 MiB. Placement would account for the selected groups' declared resources against other jobs, with the existing task-driver resource enforcement semantics.

This is a desired-state change, not just an additional placement constraint. Once the requested number of groups is selected, the other alternatives should not contribute queued allocations or prevent a deployment from completing. If fewer than the requested number of groups can be satisfied, the job must still report unmet demand. A healthy selected group should remain selected when another eligible node appears.

`count` would count distinct groups, not allocations. Selecting two groups with replica counts of two and five would require seven allocations. A partially placed group would occupy one selection slot while its remaining replicas are scheduled normally; this proposal does not require gang scheduling. Named selections would allow a job to combine independent choices with ordinary required groups.

Updates need to preserve that distinction. With a canary update, the previous selection must remain available while the new version is tried. A canary should be able to use a different alternative: an old Orin allocation can continue serving while a new Thor allocation becomes healthy. Promotion should retain the tested allocations and retire the old selection according to the update policy. Failure should preserve the previous service. Requiring both versions to use the same hardware profile would defeat the reason for the selection.

An implementation is available in #28530. It retains real task groups and the existing resource allocator, and records selection ownership and rollout transitions alongside allocations and deployments. Selection must survive a leader change and must reflect allocations accepted by the leader, including partial plan commits. It must not depend on rewriting the registered job's counts.

The first scope is service jobs. Batch completion and per-node system scheduling need separate definitions; the new block should reject unsupported job types explicitly. Jobs without a selection should retain their existing behavior.

The validation plan covers initial placement with missing profiles, partial capacity, multiple selections, replica counts greater than one, repeated evaluations, reschedule policies, drain, disconnect and preemption handling, partial plan acceptance, snapshot recovery, and stopping or purging the job. Deployment coverage includes cross-profile canaries, manual and automatic promotion, failed updates, rollback, and a newer version arriving during an update. API, CLI, and UI should report effective demand without hiding a genuinely unfilled selection. The PR records completed tests, live-stage results, and remaining validation limits separately from these acceptance criteria. Feedback on the desired-state model and its interaction with deployments would help settle the design.

### Use-cases

We run the same recognition service across a mixed fleet of x86 machines and Jetson Orin and Thor nodes. Node metadata already lets the task choose an appropriate image variant. The resource budget also differs by platform: a newer accelerator can need substantially less host CPU and memory for the same workload.

Keeping these machines in one scheduling pool lets a service use whichever suitable capacity is available, including after a node failure. Separate jobs require something outside Nomad to decide which implementation should run and to coordinate their lifecycle. A group selection would let Nomad make that choice using the resources already declared in each group.

The same need applies to alternative implementations with different device requirements or task layouts; the proposal is not tied to an architecture, driver, or metadata convention.

### Attempted Solutions

We tested a service job with three constrained groups and a job-level `distinct_property` constraint on a constant value to permit only one allocation. It does choose a feasible profile and accounts for that allocation's resources correctly. However, all three groups still have desired count one. The other two remain unplaced, the UI reports degraded status, and deployment reconciliation continues to expect them. That prevents successful native deployments without changing the desired group counts or disabling deployment tracking.

Changing the losing groups to count zero through an external controller removes that demand, but moves profile selection and recovery into a second control loop that edits the job. A device plugin alone does not make CPU and memory requests conditional or change which groups are required.

Related discussions:

- #23509, my earlier request about node-dependent jobspec values.
- #20063, a request for node-dependent values in the resources block.
- #27402 and its proposed implementation in PR #27391, alternative device reservations. Those address alternatives within a device request; this proposal concerns complete task groups with different resource and task definitions.

Contributor guide

No contributing guide indexed for this repository

Research direction

Review the implementation in PR #28530 first, then trace the existing resource allocator, deployment, allocation, and snapshot-recovery paths named in the proposal. Validate the listed placement, rollout, failure, recovery, and reporting scenarios, with completion requiring the stated desired-state behavior and explicit handling of unsupported job types.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.