etcd client freezes the entire vault when one etcd follower stops replying
- Dominant language
- Go
- Stars
- 36.3k
- Forks
- 4.8k
- PR merge metrics
- PR metrics pending
Description
When one etcd follower stops responding (e.g. during defrag) vault freezes and hangs on reads and writes.
Etcd client keeps waiting for the response from a node when it receives ACKs on the TCP requests from the kernel, but doesn't receive a response from the etcd process.
This doesn't happen when etcd is properly shut down and etcd client receives RSTs on a closed port.
### Steps to reproduce the behavior:
1. Start vault with 3-node etcd as a backend:
```hcl
backend "etcd" {
address = "https://192.168.1.10:2379,https://192.168.1.11:2379,https://192.168.1.12:2379"
ha_enabled = "true"
sync = "true"
api = "v3"
request_timeout = "15s"
lock_timeout = "20s"
}
```
3. Run `systemctl freeze etcd` (mimics what happens during long etcd defrag).
4. See that vault doesn't respond to anything and appears down. No failover happens.
5. Run `systemctl thaw etcd` to resume
6. See that vault is operational again
### Expected behavior:
etcd client correctly ignores failure of one etcd client as it does when port is closed
### Environment:
* Vault Server Version (retrieve with `vault status`): v2.0.4
* Server Operating System/Architecture: Debian 13, amd64
### Related:
https://github.com/hashicorp/vault/issues/23074
https://github.com/hashicorp/vault/issues/4961
https://github.com/hashicorp/vault/issues/9920
https://github.com/hashicorp/vault/pull/6285: request timeouts don't help because they don't evict a dead endpoint
Contributor guide
Research direction
Start by reproducing the freeze with the documented three-node etcd backend configuration, freezing one etcd process with systemctl while Vault is running. Trace the etcd backend client behavior when a TCP connection accepts requests but stops returning responses, and compare it with a closed port. Done means Vault remains responsive and fails over while one follower is unresponsive, then recovers after thawing it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100