Watches configured in the agent config are incompatible with Agent TLS when non-localhost address is used for client address
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
This bug was originally reported in #11357. Note this only applies to the [watches in the agent config](https://www.consul.io/docs/agent/options#watches). Watches [that use the CLI commad](https://www.consul.io/commands/watch) should work because the user can specify a hostname instead of an IP address.
The problem can be seen following this code path:
https://github.com/hashicorp/consul/blob/7bbad6fe560ab27f51dd6c5cd16f28418082424c/agent/config/runtime.go#L1712-L1715
https://github.com/hashicorp/consul/blob/7bbad6fe560ab27f51dd6c5cd16f28418082424c/agent/config/runtime.go#L1594-L1595
The problem is that the configured watches look up the address to dial from the agent configuration. The agent configuration must use IP address to bind, but the TLS certificates will not necessarily contain IP SANs, because IP addresses could change. They generally will contain DNS SANs, or IP SANs for `localhost 127.0.0.1`.
When localhost is used as the client address the watches work fine because the TLS certificate has those listed as IP SANs, but as soon as some interface is used for the client address, it no longer works.
**Workarounds**:
* use a localhost address as the first client addresses (**recommended**) - this is a common setup, which is probably why we have not noticed this problem before. Each of these https://www.consul.io/docs/agent/options#addresses supports a space separated list, so use `127.0.0.1 `.
* disable `verify_outgoing` - this is particularly not great because it also disables it for all RPC communication, which is not recommended
One option for fixing this problem would be to allow the user to configure a `hostname` to use for performing these watches. That way the TLS certificate would be valid. However it would require the user to make sure their DNS resolves that address correctly (possibly by using `/etc/hosts`).
Contributor guide
Research direction
Start in agent/config/runtime.go at the cited lines around 1712-1715 and 1594-1595, tracing how configured watches choose their dial address and use Agent TLS. Reproduce the failure with a non-localhost client address and outgoing verification enabled; done means configured watches work with TLS without disabling verification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- networking, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100