hashicorp / hashicorp/nomad

periodic job and `scaling` issues with the CLI

Open
#11,077 5 comments 0 reactions 0 assignees View on GitHub
theme/autoscaling type/bug type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Nomad version
Nomad v1.1.3 (8c0c8140997329136971e66e4c2337dfcf932692)

### Operating system and Environment details
Amazon Linux 2

### Issue
I have the following periodic job (stripped down and minor details like image name changed).

My use case is the following:
During my workflow, I want to `stop` (or `pause`) the "sample1" job mentioned below.
Then update some other `service` jobs.
When the `service` jobs have been updated successfully, I want to resume (or unpause) the the periodic job `sample1`.
I want to do this via the Nomad CLI to keep it simple.

#### Issue 1:
There is a `job stop`, but there is no `job start` from the existing definition as it exists. We have to re-submit the job from a job file.
(I know I could save the job definition before stopping it, but it would get messy really quick as there are many such CRON jobs)

So, I thought of using the `scaling` stanza with the periodic job to "scale it down to 0" as the "stop" equivalent, and reset the count to "1" as the "resume" equivalent.

Here begins the UI issue ...

### Reproduction steps

To `scale down` the job I do:

```
$ nomad job scale -verbose sample1 group1 0
==> No evaluation with id "" found

$ echo $?
1

```

The above command emits an error message `==> No evaluation with id "" found`

Note: the task group count has **successfully** been set to 0 (zero), just that there is an error message and a bad return value! :cry:

#### Issue 2:

Once I set the task group count to 0 (zero), I expect the Nomad to do nothing for this job, just sit there, but I notice the periodic "every two minute" job launches still going on with **no allocations** (which is the right thing)

#### Issue 3:
The `-verbose` flag does nothing, shows nothing extra! :confused:

```
$ nomad job scale -verbose sample1 group1 1
==> No evaluation with id "" found
$ echo $?
1
```

#### Expected Result

Once I set the scale to `0`, I expect the system to NOT do the periodic launches with the task group count of `0`.

#### Actual Result

The scheduler keeps launching the periodic instance of the job, but with group count set to 0.

### Job file (if appropriate)

```hcl
# bug report

job "sample1" {
type = "batch"
region = "myregion"
datacenters = ["myregion"]

periodic {
cron = "*/2 * * * * *"
prohibit_overlap = true
}

constraint {
attribute = "${node.class}"
value = "worker-node"
}

group "group1" {
count = 1

scaling {
enabled = true
min = 0
max = 5
}

task "task1" {
template {
data = <

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported behavior with the `nomad job scale -verbose sample1 group1 0` command and the supplied periodic job configuration. Investigate why scaling succeeds while reporting an empty evaluation and why periodic launches continue at a group count of zero. Done means the command reports the correct result and return value, verbose output is meaningful, and zero scaling prevents the unwanted periodic launches.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.