Constraints violated when allocation failed + rescheduled?
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
Nomad version
Nomad v1.1.0 (2678c3604bc9530014208bc167415e167fd440fc)
Operating system and Environment details
Ubuntu 18.04
Issue
We have a job with these constraints/affinities:
{"Constraints"=>
[{"LTarget"=>"${meta.fly_region}",
"Operand"=>"distinct_property",
"RTarget"=>"1"},
{"LTarget"=>"${meta.fly_region}",
"Operand"=>"set_contains_any",
"RTarget"=>"atl,cdg,gru,hkg,iad,lhr,maa,nrt,sjc,syd"}],
"Spreads"=>
[{"Weight"=>100, "Attribute"=>"${meta.fly_region}", "SpreadTarget"=>nil}],
"Affinities"=>
[{"Weight"=>50,
"LTarget"=>"${meta.fly_region}",
"Operand"=>"set_contains_any",
"RTarget"=>"atl,cdg,gru,hkg,iad,lhr,maa,nrt,sjc,syd"}]}
When the job is created or updated, it does what you'd expect (one allocation per fly_region). It's behaving weirdly when allocations fail, though, and violating the distinct_property constraint when it reschedules them. Two allocations failed over night and the replacements ended up creating a situation where 2x allocs were running on clients with fly_region == 'atl' and fly_region == 'maa'.
Reproduction steps
I can't reproduce this on purpose but it's happened at least twice with the same job spec.
Is this actually a bug or am I misunderstanding how this is supposed to work?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file, test, or entry point is named. Start by tracing Nomad's scheduler behavior for failed allocations and rescheduling, focusing on the distinct_property constraint and fly_region metadata. Reproduce the reported job configuration if possible and verify whether replacements preserve one allocation per region.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100