hashicorp / hashicorp/nomad

Region=global is not respected when submitting job in non-global region

Open
#20,571 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stage/accepted type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Nomad version

Nomad v1.6.5

Operating system and Environment details

Our setup is 3 regions: global, zone1, zone2. ACL replication and all commands from setup guides work fine, so we consider our cluster operating normally:

Name                       Address  Port  Status  Leader  Raft Version  Build  Datacenter  Region
nomad-dev-02-zone1.zone1   xxx   4648  alive   false   3             1.6.5  zone1     zone1
nomad-dev-03-zone1.zone1   xxx   4648  alive   false   3             1.6.5  zone1     zone1
nomad-dev-04-zone1.zone1   xxx   4648  alive   true    3             1.6.5  zone1     zone1
nomad-dev-02-zone2.zone2  xxx   4648  alive   false   3             1.6.5  zone2     zone2
nomad-dev-03-zone2.zone2  xxx   4648  alive   false   3             1.6.5  zone2     zone2
nomad-dev-04-zone2.zone2  xxx   4648  alive   true    3             1.6.5  zone2     zone2
nomad-dev-01-global.global   xxx   4648  alive   false   3             1.6.5  global     global
nomad-dev-01-global.global   xxx   4648  alive   false   3             1.6.5  global     global
nomad-dev-01-global.global   xxx   4648  alive   true    3             1.6.5  global     global
nomad-dev-01-global.global   xxx   4648  alive   false   3             1.6.5  global     global
Issue

When deploying job with region=global in non-global region the field region is not respected and the job is deployed to a local region. Using region=zone2 (when deploying in zone1) everything works as expected.

Reproduction steps
  1. Setup a cluster with a global region and two non-global region (for example, zone1 and zone2).
  2. Use nomad job init on a server in zone1 to create an example job. The job is created with # region = "global", but removing # doesn't fix the issue.
  3. Submit a job using CLI or API. See that the job is deployed to a local region.
  4. Change region to zone2 and see that job is deployed to zone2 region
Expected Result

The job is deployed to a global (default) region

Actual Result

The job is deployed to a local region.

Moreover, when using cli we get the following error:

nomad job run example.nomad.hcl
==> 2024-05-13T17:58:51+03:00: Monitoring evaluation "d0961634"
==> No evaluation with id "d0961634-991e-15ce-0116-d6dfa9a4633b" found

And we see that local nomad server tries to find an evaluation in global region:

[DEBUG] http: request failed: method=GET path=/v1/evaluation/d0961634-991e-15ce-0116-d6dfa9a4633b?region=global error="eval not found" code=404
nomad job status -region=global
ID                                    Type     Priority  Status          Submit Date
http                                  service  50        dead            2024-01-18T13:21:38+03:00
listener                              service  50        dead            2024-01-17T16:57:03+03:00

nomad job status -region=dev-sas
ID                       Type     Priority  Status   Submit Date
example                  service  50        running  2024-05-13T18:03:15+03:00

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with a global region and two non-global regions, then compare CLI and API submission using region="global" with submission from zone1. Trace the region and evaluation handling from job submission through evaluation lookup. Done means the job is deployed to global and CLI monitoring finds the correct evaluation without the reported 404.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.