hashicorp / hashicorp/nomad

cross-region replication across versions produces log spam on leader

Open
#18,016 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

theme/federation type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

When Nomad regions are federated, the non-authoritative regions replicate various ACL-related objects such as namespaces, auth methods, and node pools. When the authoritative region is of a version that's older than the non-authoritative region, the non-authoritative region may request objects from RPCs that don't exist on the authoritative region. While mostly harmless, this produces a bunch of error log spam on the non-authoritative region's leader:

2023-07-20T11:59:15.945-0700 [ERROR] nomad: failed to fetch node pools from authoritative region: error="rpc error: rpc: can't find service NodePool.List"
2023-07-20T11:59:15.946-0700 [ERROR] nomad: failed to fetch ACL auth-methods from authoritative region: error="rpc error: rpc: can't find method ACL.ListAuthMethods"
2023-07-20T11:59:15.953-0700 [ERROR] nomad: failed to fetch ACL binding rules from authoritative region: error="rpc error: rpc: can't find method ACL.ListBindingRules"
2023-07-20T11:59:46.014-0700 [ERROR] nomad: failed to fetch node pools from authoritative region: error="rpc error: rpc: can't find service NodePool.List"
2023-07-20T11:59:46.015-0700 [ERROR] nomad: failed to fetch ACL auth-methods from authoritative region: error="rpc error: rpc: can't find method ACL.ListAuthMethods"
2023-07-20T11:59:46.022-0700 [ERROR] nomad: failed to fetch ACL binding rules from authoritative region: error="rpc error: rpc: can't find method ACL.ListBindingRules"
2023-07-20T12:00:16.083-0700 [ERROR] nomad: failed to fetch ACL auth-methods from authoritative region: error="rpc error: rpc: can't find method ACL.ListAuthMethods"
2023-07-20T12:00:16.084-0700 [ERROR] nomad: failed to fetch node pools from authoritative region: error="rpc error: rpc: can't find service NodePool.List"
2023-07-20T12:00:16.090-0700 [ERROR] nomad: failed to fetch ACL binding rules from authoritative region: error="rpc error: rpc: can't find method ACL.ListBindingRules"

It'd be nice if we could do version detection here, the same way we do for RPCs within a region, and then decide not to send these RPCs until the authoritative region is up-to-date as well.

Originally reported in https://github.com/hashicorp/nomad/issues/18007#issuecomment-1644445673

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace the cross-region ACL and node-pool replication paths that emit the listed errors, then inspect how version detection is handled for intra-region RPCs. Done means unsupported RPCs are not sent to an older authoritative region, and the repeated error log spam is eliminated; the issue names no files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend-api-design, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.