Scaling a (spread) job down appears to fully recreate allocs
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
I’ve noticed that when using `nomad job scale `, that the number of placed/healthy allocations seems to restart from 0; specifically I've noticed this when running a job with a `spread` block with differing percents and lowering to a count where those percents are not equally distributable.
I've been running with the following jobspec on 2 nodes which each have a `year` metadata property. In testing please feel free to modify this however you see fit.
I have a sneaking suspicion that because my 70/30 spread split makes sense when I have `count=10` (7 and 3 allocs on my nodes), but is no longer strictly possible when moving to `count=2`.
```
job "hello-world" {
# Specifies the datacenters within which this job should be run.
# Leave as "dc1" for the default datacenter.
datacenters = ["dc1"]
meta {
# User-defined key/value pairs that can be used in your jobs.
# You can also use this meta block within Group and Task levels.
foo = "bar"
}
# A group defines a series of tasks that should be co-located
# on the same client (host). All tasks within a group will be
# placed on the same host.
group "servers" {
count = 10
spread {
attribute = "${meta.year}"
weight = 100
target "2023" {
percent = 70
}
target "2017" {
percent = 30
}
}
network {
port "www" {
to = 8001
}
}
service {
provider = "nomad"
port = "www"
}
# Tasks are individual units of work that are run by Nomad.
task "web" {
# This particular task starts a simple web server within a Docker container
driver = "docker"
config {
image = "busybox:1"
command = "httpd"
args = ["-v", "-f", "-p", "${NOMAD_PORT_www}", "-h", "/local"]
ports = ["www"]
}
template {
data = <Hello, Nomad!
- Task: {{env "NOMAD_TASK_NAME"}}
- Group: {{env "NOMAD_GROUP_NAME"}}
- Job: {{env "NOMAD_JOB_NAME"}}
- Metadata value for foo: {{env "NOMAD_META_foo"}}
- Currently running on port: {{env "NOMAD_PORT_www"}}
EOF
destination = "local/index.html"
}
# Specify the maximum resources required to run the task
resources {
cpu = 50
memory = 64
}
}
}
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the behavior with the supplied two-node jobspec, its 70/30 spread block, and the `nomad job scale` command while reducing the count from 10 to 2. Trace the scaling and spread-placement entry points to determine why existing allocations restart; done means scaling down preserves healthy allocations where possible while applying a valid spread.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100