Region=global is not respected when submitting job in non-global region
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
Nomad version
Nomad v1.6.5
Operating system and Environment details
Our setup is 3 regions: global, zone1, zone2. ACL replication and all commands from setup guides work fine, so we consider our cluster operating normally:
Name Address Port Status Leader Raft Version Build Datacenter Region
nomad-dev-02-zone1.zone1 xxx 4648 alive false 3 1.6.5 zone1 zone1
nomad-dev-03-zone1.zone1 xxx 4648 alive false 3 1.6.5 zone1 zone1
nomad-dev-04-zone1.zone1 xxx 4648 alive true 3 1.6.5 zone1 zone1
nomad-dev-02-zone2.zone2 xxx 4648 alive false 3 1.6.5 zone2 zone2
nomad-dev-03-zone2.zone2 xxx 4648 alive false 3 1.6.5 zone2 zone2
nomad-dev-04-zone2.zone2 xxx 4648 alive true 3 1.6.5 zone2 zone2
nomad-dev-01-global.global xxx 4648 alive false 3 1.6.5 global global
nomad-dev-01-global.global xxx 4648 alive false 3 1.6.5 global global
nomad-dev-01-global.global xxx 4648 alive true 3 1.6.5 global global
nomad-dev-01-global.global xxx 4648 alive false 3 1.6.5 global global
Issue
When deploying job with region=global in non-global region the field region is not respected and the job is deployed to a local region. Using region=zone2 (when deploying in zone1) everything works as expected.
Reproduction steps
- Setup a cluster with a global region and two non-global region (for example, zone1 and zone2).
- Use
nomad job initon a server in zone1 to create an example job. The job is created with# region = "global", but removing#doesn't fix the issue. - Submit a job using CLI or API. See that the job is deployed to a local region.
- Change region to
zone2and see that job is deployed tozone2region
Expected Result
The job is deployed to a global (default) region
Actual Result
The job is deployed to a local region.
Moreover, when using cli we get the following error:
nomad job run example.nomad.hcl
==> 2024-05-13T17:58:51+03:00: Monitoring evaluation "d0961634"
==> No evaluation with id "d0961634-991e-15ce-0116-d6dfa9a4633b" found
And we see that local nomad server tries to find an evaluation in global region:
[DEBUG] http: request failed: method=GET path=/v1/evaluation/d0961634-991e-15ce-0116-d6dfa9a4633b?region=global error="eval not found" code=404
nomad job status -region=global
ID Type Priority Status Submit Date
http service 50 dead 2024-01-18T13:21:38+03:00
listener service 50 dead 2024-01-17T16:57:03+03:00
nomad job status -region=dev-sas
ID Type Priority Status Submit Date
example service 50 running 2024-05-13T18:03:15+03:00
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with a global region and two non-global regions, then compare CLI and API submission using region="global" with submission from zone1. Trace the region and evaluation handling from job submission through evaluation lookup. Done means the job is deployed to global and CLI monitoring finds the correct evaluation without the reported 404.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100