hashicorp / hashicorp/consul

Service ACL token used to perform blocked queries on behalf of a connect service is not refreshed

Open
#11,949 0 comments 1 reaction 0 assignees View on GitHub
theme/connect
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

When a connect service is created on a cluster with ACL activated, The token used to create the service is stored in the agent and used for all the blocking queries related to that service. This token cannot be changed and if the token is deleted the service need to be recreated with a new token to continue to function properly.

Also a token is configured in envoy and provided as part of the envoy XDS requests, this token is authenticated and proper access privilege is checked for it but is not used for any agent to server operations related to that service.

#### Reproduction Steps

The issue that could happen is as follow:

- a service is created with a token A
- an envoy is bootstrapped with the same token A
- At this point everything work correctly
- after a while, the service operator decide to rotate the token and create token B with the same policies
- envoy config for the service is updated with token B
- token A is deleted

At this point, when consul agent receive an XDS request from envoy it will authenticate token B (which is present and have the correct privileges). When the agent create blocking queries to update the service (for example leaf certificates) it will use token A which will be denied by the server (because token A is deleted). This lead to a service that never receive updates.

This is also true in the use case where customers use Nomad and Consul, the Nomad/Consul integration permit to Nomad to handle the lifecycle of Consul ACL tokens (Create, inject to envoy and delete), a discrepancy between what is injected in envoy and what the service have as a token could lead to the same issue.

#### Proposed Fix

Every time, consul agent receive a request from envoy with an ACL token the agent will refresh the service token with it to ensure that the same token is used to perform blocking queries on behalf of the service.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how the Consul agent handles authenticated Envoy XDS requests and how it creates blocking queries for service updates. Compare the token received from Envoy with the token retained for agent-to-server operations, then verify that rotating and deleting the original token still allows updates using the replacement token.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
authentication, backend, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.