hashicorp / hashicorp/consul

healthcheck token impersonation issue ("sunken tokens")

Open
#12,145 5 comments 2 reactions 0 assignees View on GitHub
theme/consul-nomad theme/health-checks theme/operator-usability type/enhancement
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

This initial error report aims at getting feedback from developers to put me into a position where I can better pin down the error.

consul: v1.10.3

In a given cluster (ceteris paribus), we have the following commands run from **within a nomad allocation:**

- A
```console
curl -XPUT $CONSUL_HTTP_ADDR/v1/agent/service/register \
-H 'Content-Type: application/json' -d '{"name":"testnet-staging-database","id": "testnet-staging-database/pg-2", "address": "10.32.153.145", "port": 5432, "check": { "interval": "5s", "timeout": "2s", "DeregisterCriticalServiceAfter": "150.0s", "http": "https://10.32.153.145:8008/replica"}, "tags": [ "replica" ], "enable_tag_override": true }'
```
- B
```console
curl -XPUT $CONSUL_HTTP_ADDR/v1/agent/service/register?token=$CONSUL_HTTP_TOKEN \
-H 'Content-Type: application/json' -d '{"name":"testnet-staging-database","id": "testnet-staging-database/pg-2", "address": "10.32.153.145", "port": 5432, "check": { "interval": "5s", "timeout": "2s", "DeregisterCriticalServiceAfter": "150.0s", "http": "https://10.32.153.145:8008/replica"}, "tags": [ "replica" ], "enable_tag_override": true }'
```

_The nomad allocation recovers a token lease (`CONSUL_HTTP_TOKEN`) from `{{ with secret "consul/creds/patroni" }}` on startup._

- A produces an agent state where the agent can update the server health-check _at all times_
- B produces an agent state where the agent can NOT update the server health-check _at all times_
- Specifically, the consul-agent cannot update the servers any more after the token lease has been expired.
- Nomad appears to be managing the token lease lifecycle, that is, freeing the lease when the job is stopped.
- Note!!: agent has correct state (health-check: `critical`) when job stops. Server has not.

It appears as if the agent's healthcheck would inherit the token lease from the initiating party which breaks the purpose of healthchecks notification if that token does not outlive the initiating party.

This is critical since an effectively dead service instance is never updated againts the cluster and hence the cluster effectively serves invalid DNS responses.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.