cross-region replication across versions produces log spam on leader
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
When Nomad regions are federated, the non-authoritative regions replicate various ACL-related objects such as namespaces, auth methods, and node pools. When the authoritative region is of a version that's older than the non-authoritative region, the non-authoritative region may request objects from RPCs that don't exist on the authoritative region. While mostly harmless, this produces a bunch of error log spam on the non-authoritative region's leader:
2023-07-20T11:59:15.945-0700 [ERROR] nomad: failed to fetch node pools from authoritative region: error="rpc error: rpc: can't find service NodePool.List"
2023-07-20T11:59:15.946-0700 [ERROR] nomad: failed to fetch ACL auth-methods from authoritative region: error="rpc error: rpc: can't find method ACL.ListAuthMethods"
2023-07-20T11:59:15.953-0700 [ERROR] nomad: failed to fetch ACL binding rules from authoritative region: error="rpc error: rpc: can't find method ACL.ListBindingRules"
2023-07-20T11:59:46.014-0700 [ERROR] nomad: failed to fetch node pools from authoritative region: error="rpc error: rpc: can't find service NodePool.List"
2023-07-20T11:59:46.015-0700 [ERROR] nomad: failed to fetch ACL auth-methods from authoritative region: error="rpc error: rpc: can't find method ACL.ListAuthMethods"
2023-07-20T11:59:46.022-0700 [ERROR] nomad: failed to fetch ACL binding rules from authoritative region: error="rpc error: rpc: can't find method ACL.ListBindingRules"
2023-07-20T12:00:16.083-0700 [ERROR] nomad: failed to fetch ACL auth-methods from authoritative region: error="rpc error: rpc: can't find method ACL.ListAuthMethods"
2023-07-20T12:00:16.084-0700 [ERROR] nomad: failed to fetch node pools from authoritative region: error="rpc error: rpc: can't find service NodePool.List"
2023-07-20T12:00:16.090-0700 [ERROR] nomad: failed to fetch ACL binding rules from authoritative region: error="rpc error: rpc: can't find method ACL.ListBindingRules"
It'd be nice if we could do version detection here, the same way we do for RPCs within a region, and then decide not to send these RPCs until the authoritative region is up-to-date as well.
Originally reported in https://github.com/hashicorp/nomad/issues/18007#issuecomment-1644445673
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace the cross-region ACL and node-pool replication paths that emit the listed errors, then inspect how version detection is handled for intra-region RPCs. Done means unsupported RPCs are not sent to an older authoritative region, and the repeated error log spam is eliminated; the issue names no files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend-api-design, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100