hashicorp / hashicorp/vault

etcd client freezes the entire vault when one etcd follower stops replying

Open
#32,091 0 comments 1 reaction 0 assignees View on GitHub
bug core/ha storage/etcd
Dominant language
Go
Stars
36.3k
Forks
4.8k
PR merge metrics
PR metrics pending

Description

When one etcd follower stops responding (e.g. during defrag) vault freezes and hangs on reads and writes.
Etcd client keeps waiting for the response from a node when it receives ACKs on the TCP requests from the kernel, but doesn't receive a response from the etcd process.
This doesn't happen when etcd is properly shut down and etcd client receives RSTs on a closed port.

### Steps to reproduce the behavior:
1. Start vault with 3-node etcd as a backend:
```hcl
backend "etcd" {
address = "https://192.168.1.10:2379,https://192.168.1.11:2379,https://192.168.1.12:2379"
ha_enabled = "true"
sync = "true"
api = "v3"
request_timeout = "15s"
lock_timeout = "20s"
}
```
3. Run `systemctl freeze etcd` (mimics what happens during long etcd defrag).
4. See that vault doesn't respond to anything and appears down. No failover happens.
5. Run `systemctl thaw etcd` to resume
6. See that vault is operational again

### Expected behavior:

etcd client correctly ignores failure of one etcd client as it does when port is closed

### Environment:

* Vault Server Version (retrieve with `vault status`): v2.0.4
* Server Operating System/Architecture: Debian 13, amd64

### Related:

https://github.com/hashicorp/vault/issues/23074
https://github.com/hashicorp/vault/issues/4961
https://github.com/hashicorp/vault/issues/9920

https://github.com/hashicorp/vault/pull/6285: request timeouts don't help because they don't evict a dead endpoint

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the freeze with the documented three-node etcd backend configuration, freezing one etcd process with systemctl while Vault is running. Trace the etcd backend client behavior when a TCP connection accepts requests but stops returning responses, and compare it with a closed port. Done means Vault remains responsive and fails over while one follower is unresponsive, then recovers after thawing it.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.