hashicorp / hashicorp/nomad

Constraints violated when allocation failed + rescheduled?

Open
#11,199 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

stage/needs-investigation theme/restart/reschedule theme/scheduling type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Nomad version

Nomad v1.1.0 (2678c3604bc9530014208bc167415e167fd440fc)

Operating system and Environment details

Ubuntu 18.04

Issue

We have a job with these constraints/affinities:

{"Constraints"=>
  [{"LTarget"=>"${meta.fly_region}",
    "Operand"=>"distinct_property",
    "RTarget"=>"1"},
   {"LTarget"=>"${meta.fly_region}",
    "Operand"=>"set_contains_any",
    "RTarget"=>"atl,cdg,gru,hkg,iad,lhr,maa,nrt,sjc,syd"}],
 "Spreads"=>
  [{"Weight"=>100, "Attribute"=>"${meta.fly_region}", "SpreadTarget"=>nil}],
 "Affinities"=>
  [{"Weight"=>50,
    "LTarget"=>"${meta.fly_region}",
    "Operand"=>"set_contains_any",
    "RTarget"=>"atl,cdg,gru,hkg,iad,lhr,maa,nrt,sjc,syd"}]}

When the job is created or updated, it does what you'd expect (one allocation per fly_region). It's behaving weirdly when allocations fail, though, and violating the distinct_property constraint when it reschedules them. Two allocations failed over night and the replacements ended up creating a situation where 2x allocs were running on clients with fly_region == 'atl' and fly_region == 'maa'.

Reproduction steps

I can't reproduce this on purpose but it's happened at least twice with the same job spec.

Is this actually a bug or am I misunderstanding how this is supposed to work?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file, test, or entry point is named. Start by tracing Nomad's scheduler behavior for failed allocations and rescheduling, focusing on the distinct_property constraint and fly_region metadata. Reproduce the reported job configuration if possible and verify whether replacements preserve one allocation per region.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.