hashicorp / hashicorp/nomad

Scaling a (spread) job down appears to fully recreate allocs

Open
#16,161 0 comments 0 reactions 0 assignees View on GitHub
stage/accepted theme/autoscaling theme/scheduling type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

I’ve noticed that when using `nomad job scale `, that the number of placed/healthy allocations seems to restart from 0; specifically I've noticed this when running a job with a `spread` block with differing percents and lowering to a count where those percents are not equally distributable.

I've been running with the following jobspec on 2 nodes which each have a `year` metadata property. In testing please feel free to modify this however you see fit.

I have a sneaking suspicion that because my 70/30 spread split makes sense when I have `count=10` (7 and 3 allocs on my nodes), but is no longer strictly possible when moving to `count=2`.

```
job "hello-world" {
# Specifies the datacenters within which this job should be run.
# Leave as "dc1" for the default datacenter.
datacenters = ["dc1"]

meta {
# User-defined key/value pairs that can be used in your jobs.
# You can also use this meta block within Group and Task levels.
foo = "bar"
}

# A group defines a series of tasks that should be co-located
# on the same client (host). All tasks within a group will be
# placed on the same host.
group "servers" {

count = 10
spread {
attribute = "${meta.year}"
weight = 100

target "2023" {
percent = 70
}

target "2017" {
percent = 30
}
}

network {
port "www" {
to = 8001
}
}

service {
provider = "nomad"
port = "www"
}

# Tasks are individual units of work that are run by Nomad.
task "web" {
# This particular task starts a simple web server within a Docker container
driver = "docker"

config {
image = "busybox:1"
command = "httpd"
args = ["-v", "-f", "-p", "${NOMAD_PORT_www}", "-h", "/local"]
ports = ["www"]
}

template {
data = <Hello, Nomad!


  • Task: {{env "NOMAD_TASK_NAME"}}

  • Group: {{env "NOMAD_GROUP_NAME"}}

  • Job: {{env "NOMAD_JOB_NAME"}}

  • Metadata value for foo: {{env "NOMAD_META_foo"}}

  • Currently running on port: {{env "NOMAD_PORT_www"}}


EOF
destination = "local/index.html"
}

# Specify the maximum resources required to run the task
resources {
cpu = 50
memory = 64
}
}
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the behavior with the supplied two-node jobspec, its 70/30 spread block, and the `nomad job scale` command while reducing the count from 10 to 2. Trace the scaling and spread-placement entry points to determine why existing allocations restart; done means scaling down preserves healthy allocations where possible while applying a valid spread.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.