hashicorp / hashicorp/nomad

Job is "running" with no allocations. Is it expected?

Open
#10,359 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stage/accepted theme/scheduling type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Nomad version

Nomad v1.0.4 (9294f35f9aa8dbb4acb6e85fa88e3e2534a3e41a)

Operating system and Environment details

Centos 7

Issue

I started type=system job and it reports like a running without any allocations.
I still try to figure out what is wrong with my definition but I wonder if that is expected from the presentation\UX point of view.

$nomad status -verbose -namespace=system vector
ID            = vector
Name          = vector
Submit Date   = 2021-04-12T10:11:07Z
Type          = system
Priority      = 50
Datacenters   = test-run-kproskurin-rm
Namespace     = system
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
vector      0       0         0        0       0         0

Evaluations
ID                                    Priority  Triggered By  Status    Placement Failures
192438bb-720d-b2b5-167f-cf0127ba745c  50        job-register  complete  false

Allocations
No allocations placed
Reproduction steps
Expected Result

No allocation == not running?..

Actual Result

No allocation == running

Job file

https://gist.github.com/Oloremo/ce78aae9b1957e06db067f97628fa6d4

UPD:
Ok so the issue is the following:

If you specify a non-existent dc in the system job definition it will be "created":
image (2)

And allocation will be in the state described above.

So here we observe at least a few issues:

  1. Nomad's actions in case a non-existent(not defined in agent configuration) datacenter configuration in a job. Imo it should fail during the plan.
  2. Job is started but the scheduler is unable to allocate it but there is no error or any problem indication.
  3. Seems like all that related only to the system jobs.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the linked job definition and checking the nomad status -verbose output for a system job targeting a nonexistent datacenter. Compare the reported job, evaluation, and allocation states with the expected behavior described in the issue. Done means the handling and user-visible status of this case are clearly resolved for the reported scenarios.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.