Service ACL token used to perform blocked queries on behalf of a connect service is not refreshed
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
When a connect service is created on a cluster with ACL activated, The token used to create the service is stored in the agent and used for all the blocking queries related to that service. This token cannot be changed and if the token is deleted the service need to be recreated with a new token to continue to function properly.
Also a token is configured in envoy and provided as part of the envoy XDS requests, this token is authenticated and proper access privilege is checked for it but is not used for any agent to server operations related to that service.
#### Reproduction Steps
The issue that could happen is as follow:
- a service is created with a token A
- an envoy is bootstrapped with the same token A
- At this point everything work correctly
- after a while, the service operator decide to rotate the token and create token B with the same policies
- envoy config for the service is updated with token B
- token A is deleted
At this point, when consul agent receive an XDS request from envoy it will authenticate token B (which is present and have the correct privileges). When the agent create blocking queries to update the service (for example leaf certificates) it will use token A which will be denied by the server (because token A is deleted). This lead to a service that never receive updates.
This is also true in the use case where customers use Nomad and Consul, the Nomad/Consul integration permit to Nomad to handle the lifecycle of Consul ACL tokens (Create, inject to envoy and delete), a discrepancy between what is injected in envoy and what the service have as a token could lead to the same issue.
#### Proposed Fix
Every time, consul agent receive a request from envoy with an ACL token the agent will refresh the service token with it to ensure that the same token is used to perform blocking queries on behalf of the service.
Contributor guide
Research direction
Start by tracing how the Consul agent handles authenticated Envoy XDS requests and how it creates blocking queries for service updates. Compare the token received from Envoy with the token retained for agent-to-server operations, then verify that rotating and deleting the original token still allows updates using the replacement token.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- authentication, backend, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100