Confusion about removed task groups
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Nomad version
$ nomad version
Nomad v0.11.2+ent (7e7a895f627eaf94d1f2e63634b035df02a04be2
### Issue
When task groups are removed during the lifetime of a job, the removed task groups are still displayed in the CLI and are included in the job summary metrics but are not visible in the UI.
This gets messy quickly - the CLI/UI discrepancy is confusing but the real pain are the job summary metrics as they make alerting rather ... difficult.
Ideally, the UI and CLI should **always** give the same information and job summary metrics should **NOT** be generated for task groups that have been explicitly removed from a jobfile.
### Reproduction steps
1. Deploy the jobfile below containing two task groups.
```
$ nomad plan foo.nomad
+ Job: "foo"
+ Task Group: "first_group" (1 create)
+ Task: "redis" (forces create)
+ Task Group: "second_group" (1 create)
+ Task: "redis" (forces create)
Scheduler dry-run:
- All tasks successfully allocated.
Job Modify Index: 0
To submit the job with version verification run:
nomad job run -check-index 0 foo.nomad
When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.
```
After deploying the job :
- `nomad job status foo` will display both groups
- the UI will display both task groups
- job summary metrics will be generated for both task groups.
2. Once deployed, remove the task group `second_group` from the jobfile and run the job again.
```
$ nomad plan foo.nomad
+/- Job: "foo"
Task Group: "first_group" (1 in-place update)
Task: "redis"
- Task Group: "second_group" (1 destroy)
- Task: "redis" (forces destroy)
Scheduler dry-run:
- All tasks successfully allocated.
Job Modify Index: 798339
To submit the job with version verification run:
nomad job run -check-index 798339 foo.nomad
When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.
```
- `nomad job status foo` will still display both groups
- the UI will only display the remaining, active task group
- job summary metrics will still be generated for both task groups.
### Job file (if appropriate)
```
job "foo" {
datacenters = ["dc1"]
group "first_group" {
task "redis" {
driver = "docker"
config {
image = "redis:3.2"
port_map {
db = 6379
}
}
resources {
cpu = 500
memory = 256
network {
mbits = 10
port "db" {}
}
}
}
}
group "second_group" {
task "redis" {
driver = "docker"
config {
image = "redis:3.2"
port_map {
db = 6379
}
}
resources {
cpu = 500
memory = 256
network {
mbits = 10
port "db" {}
}
}
}
}
}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the lifecycle described with the provided Nomad job file, comparing `nomad job status foo`, the UI, and job summary metrics before and after removing `second_group`. Trace the paths responsible for task-group status and summary metrics; done means removed groups no longer appear in CLI output or metrics and the CLI and UI agree.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- backend, cli, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100