Cross-job anti-colocation constraints and affinities
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Proposal
I want to be able to specify constraints and affinities across jobs, in particular an operator like `distinct_hosts` across tasks in different job specs within the same namespace. I would like to be able to specify these cross-jobspec operators both as a hard constraint and as affinity for best effort.
### Use cases
I want to run sets of related but independent jobs that are run on distinct hosts or in distinct datacenters. Mostly so that a host or datacenter failure doesn’t take out multiple apps within the same group (for example if they are replicas). Or to be able to define two groups of apps as anti-colocated with each other, to make sure a single host failure cannot affect both groups. And to avoid having certain apps placed on the same host, as they could be noisy neighbors.
### Related issues
Cross-job placement has come up before. For example \#11066 (and \#24610) asks for cross-job co-location. Another related issue, \#11197, asks for count limits on cross-jobs sharing a meta value which is another form of cross-job scheduling decisions. However, neither covers the ability to specify cross-job anti-colocation.
### Prior art
This proposal is very similar to the Kubernetes pod affinity rules as defined by `podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution` and `podAntiAffinity.preferredDuringSchedulingIgnoredDuringExecution` as outlined in https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/.
### Attempted Solutions
Nomad already provides `distinct_hosts` and `distinct_property` operators within a job spec. These are the semantics I want, but they only apply within a single job spec, and my apps are each defined in their own job spec.
I prototyped this in Nomad by representing placement groups via job spec meta entries, and introducing a new constraint operand, `distinct_job_meta`, that makes a node infeasible if it already runs a non-terminal allocation of any other job in the same namespace carrying the same meta entry. I introduced a similar `colocated_job_meta` affinity operand to provide the best-effort variant (only allows negative weights). Datacenter anti-colocation was represented with similar `distinct_job_meta_dc` / `colocated_job_meta_dc` operands.
In HCL this would look like:
```
job "service-a" {
meta {
placement_group = "group-1"
}
constraint {
attribute = "placement_group"
operator = "distinct_job_meta"
value = "group-1"
}
}
```
Due to Nomad's optimistic concurrent planning, I implemented validation at the serialized plan applier (since two concurrent plans could be independently feasible and yet conflict), similar to how the existing `AllocsFit` check is done on the leader.
One trade-off is that the hard constraint does mean an evaluation blocks when a group cannot place (in the common case groups are likely small relative to the number of clients, so it would be rare), and the affinity variant covers cases where best effort is acceptable.
I kept things simpler by making a job's own allocations never count against its own constraint, so members can always update or restart in place, and so it can compose with `distinct_hosts` / `distinct_property`.
If the maintainers think this is a good direction, I would be happy to continue work on it and upstream it. I also welcome any thoughts on the approach or on issues I may not have considered.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing Nomad's existing distinct_hosts and distinct_property operators and the AllocsFit check in the serialized plan applier. Compare the proposed hard-constraint and best-effort affinity semantics with the Kubernetes podAntiAffinity reference and the prototype described here. Done means cross-job, same-namespace placement can enforce distinct hosts or datacenters, or prefer separation, without conflicts from concurrent plans.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100