hashicorp / hashicorp/nomad

Confusion about removed task groups

Open
#8,025 2 comments 0 reactions 0 assignees View on GitHub
theme/ui type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
$ nomad version
Nomad v0.11.2+ent (7e7a895f627eaf94d1f2e63634b035df02a04be2

### Issue
When task groups are removed during the lifetime of a job, the removed task groups are still displayed in the CLI and are included in the job summary metrics but are not visible in the UI.

This gets messy quickly - the CLI/UI discrepancy is confusing but the real pain are the job summary metrics as they make alerting rather ... difficult.

Ideally, the UI and CLI should **always** give the same information and job summary metrics should **NOT** be generated for task groups that have been explicitly removed from a jobfile.

### Reproduction steps
1. Deploy the jobfile below containing two task groups.

```
$ nomad plan foo.nomad
+ Job: "foo"
+ Task Group: "first_group" (1 create)
+ Task: "redis" (forces create)

+ Task Group: "second_group" (1 create)
+ Task: "redis" (forces create)

Scheduler dry-run:
- All tasks successfully allocated.

Job Modify Index: 0
To submit the job with version verification run:

nomad job run -check-index 0 foo.nomad

When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.
```

After deploying the job :

- `nomad job status foo` will display both groups
- the UI will display both task groups
- job summary metrics will be generated for both task groups.

2. Once deployed, remove the task group `second_group` from the jobfile and run the job again.

```
$ nomad plan foo.nomad
+/- Job: "foo"
Task Group: "first_group" (1 in-place update)
Task: "redis"

- Task Group: "second_group" (1 destroy)
- Task: "redis" (forces destroy)

Scheduler dry-run:
- All tasks successfully allocated.

Job Modify Index: 798339
To submit the job with version verification run:

nomad job run -check-index 798339 foo.nomad

When running the job with the check-index flag, the job will only be run if the
job modify index given matches the server-side version. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.
```

- `nomad job status foo` will still display both groups
- the UI will only display the remaining, active task group
- job summary metrics will still be generated for both task groups.

### Job file (if appropriate)
```
job "foo" {
datacenters = ["dc1"]

group "first_group" {
task "redis" {
driver = "docker"

config {
image = "redis:3.2"

port_map {
db = 6379
}
}

resources {
cpu = 500
memory = 256

network {
mbits = 10
port "db" {}
}
}
}
}

group "second_group" {
task "redis" {
driver = "docker"

config {
image = "redis:3.2"

port_map {
db = 6379
}
}

resources {
cpu = 500
memory = 256

network {
mbits = 10
port "db" {}
}
}
}
}
}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the lifecycle described with the provided Nomad job file, comparing `nomad job status foo`, the UI, and job summary metrics before and after removing `second_group`. Trace the paths responsible for task-group status and summary metrics; done means removed groups no longer appear in CLI output or metrics and the CLI and UI agree.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
backend, cli, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.