hashicorp / hashicorp/nomad

client RPC fails to validate new ACL token

Open
#17,834 1 comment 0 reactions 0 assignees View on GitHub
hcc/hcp theme/auth type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

We had an internal report of a problem where the [Restart Allocation API](https://developer.hashicorp.com/nomad/api-docs/allocations#restart-allocation) would sporadically fail with the error "Unexpected response code: 403 (ACL token not found)". After some investigation, we learned that the ACL token was obtained from Vault's Nomad secrets engine shortly before the API call.

The Restart Allocation API is a "client RPC" that gets forwarded from the server to the client for execution locally. The handlers for these RPCs resolve the auth token as either an ACL token or a Workload Identity in [`resolveTokenValue` (ref `client/acl.go#L139-L178`)](https://github.com/hashicorp/nomad/blob/main/client/acl.go#L139-L178), which checks a local cache first and if the token isn't there (or the cache is expired) calls the `ACL.WhoAmI` RPC so the server can validate the token. This RPC is make with `AllowStale = true`, which means it can be served by followers without being forwarded to the leader. That was done to reduce leader load way back in https://github.com/hashicorp/nomad/commit/e9790c63b41428f3912838f9ff216d5f4307f7c6 which shipped in Nomad 0.6.3.

This appears to create a narrow race condition:
* The Vault secrets engine writes a token to Nomad's leader and returns it to the user.
* The user uses this token for Restart Allocation request.
* The Restart Allocation request hits server A (either the leader or an up-to-date follower; we'll assume it's the leader here).
* Server A authorizes the request successfully.
* Server A looks up the node with the alloc and find it's connected to server B, which is behind in replication of the ACL token.
* Server A forwards to server B.
* Server B forwards to the client.
* The client sends `ACL.WhoAmI` w/ `AllowStale = true` to server B.
* Server B fails to authenticate the request.

The only way this sequence of events seems plausible is if server B has just joined the cluster and is serving RPCs before it has completed its snapshot (see also https://github.com/hashicorp/nomad/issues/15560), or if one server is lagging far behind in replication.

The temporary workaround for our internal user is to ensure that the token written by Vault has had time to propogate before using it; the workload is being automated and is fairly aggressive about issuing a new token and then immediately using it. We're also asking the internal user to verify that the ACL token hasn't been revoked by Vault too early and check with the infra team that replication is healthy.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with client/acl.go#L139-L178 and trace resolveTokenValue through the client RPC path, then inspect the ACL.WhoAmI call and its AllowStale behavior. The issue describes a replication race but does not specify a fix, reproducer, or regression test, so the intended completion criteria need to be established first.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, authentication, distributed-systems, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.