Deadlock / cluster stall after Leadership Transfer
- Dominant language
- Go
- Stars
- 9.1k
- Forks
- 1.1k
- Avg merge
- 3h 27m
- Merged PRs (30d)
- 2
Description
Hello,
When testing this library with [Antithesis](https://antithesis.com/), we encountered a bug where a healthy cluster is unable to make progress after a leadership transfer operation that causes a goroutine to get stuck.
Version tested: 4c8f61ac9255bb95fb3b8319dfcf0ae53ab325b6
## Impact
**Liveness** - A healthy cluster is unable to commit new entries unless the affected leader is rebooted.
## Triggers
1. A leadership transfer from some node A to node B, when B's logs are not fully in sync, and the transfer is interrupted by node A having to step-down as a leader for some other reason (e.g., lease timeout).
2. A subsequent re-election of node A as the leader.
## Symptom
The re-elected leader (node A in the above example) refuses to service client requests with the error message "leadership transfer in progress"
## Reproducing
The bug is hard to reproduce exactly without Antithesis, but the general approach we followed in our test environment was as follows:
- **State machine**: Simple chain-of-blocks (the FSM maintains a running hash of all input commands received)
- **Cluster**: 3 nodes, no membership changes
- **Client**: Random workload of Apply commands intermixed with occasional Leadership Transfer commands.
- **Fault Injection**: Network partitions, node restarts
- **Test oracles**: Check for Raft safety invariants from the paper at all times + check liveness of commit progress / FSM convergence when fault injection is turned off.
### Sample Logs from a 3-node test
Most logs are from `raft`, whereas the last line below is from an [Antithesis `eventually` test command](https://antithesis.com/docs/product/test_templates/test_composer_reference/#eventually-command) that checks for the cluster's ability to make commits when fault injection is stopped and the cluster is fully reachable via the network.
```
[...]
2026-07-02T00:08:27.504Z [INFO] snapshot: creating new snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950907504.tmp
2026-07-02T00:08:27.504Z [INFO] snapshot: reaping snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950885935
2026-07-02T00:08:27.504Z [INFO] node-0: no logs to truncate
2026-07-02T00:08:27.504Z [INFO] node-0: snapshot complete up to: index=3
2026-07-02T00:08:32.868Z [ERROR] node-2: failed to take snapshot: error="leadership transfer in progress"
2026-07-02T00:08:36.448Z [INFO] node-0: starting snapshot up to: index=3
2026-07-02T00:08:36.448Z [INFO] snapshot: creating new snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950916448.tmp
2026-07-02T00:08:36.449Z [INFO] snapshot: reaping snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950894124
2026-07-02T00:08:36.449Z [INFO] node-0: no logs to truncate
2026-07-02T00:08:36.449Z [INFO] node-0: snapshot complete up to: index=3
2026-07-02T00:08:39.537Z [ERROR] node-2: failed to take snapshot: error="leadership transfer in progress"
2026-07-02T00:08:43.792Z [INFO] node-0: starting snapshot up to: index=3
2026-07-02T00:08:43.792Z [INFO] snapshot: creating new snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950923792.tmp
2026-07-02T00:08:43.792Z [INFO] snapshot: reaping snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950902333
2026-07-02T00:08:43.792Z [INFO] node-0: no logs to truncate
2026-07-02T00:08:43.792Z [INFO] node-0: snapshot complete up to: index=3
2026-07-02T00:08:49.300Z [ERROR] node-2: failed to take snapshot: error="leadership transfer in progress"
2026-07-02T00:08:52.752Z [INFO] node-0: starting snapshot up to: index=3
2026-07-02T00:08:52.752Z [INFO] snapshot: creating new snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950932752.tmp
2026-07-02T00:08:52.752Z [INFO] snapshot: reaping snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950907504
2026-07-02T00:08:52.752Z [INFO] node-0: no logs to truncate
2026-07-02T00:08:52.752Z [INFO] node-0: snapshot complete up to: index=3
2026-07-02T00:08:57.359Z [ERROR] node-2: failed to take snapshot: error="leadership transfer in progress"
2026-07-02T00:08:59.551Z [INFO] node-0: starting snapshot up to: index=3
2026-07-02T00:08:59.551Z [INFO] snapshot: creating new snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950939551.tmp
2026-07-02T00:08:59.551Z [INFO] snapshot: reaping snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950916448
2026-07-02T00:08:59.551Z [INFO] node-0: no logs to truncate
2026-07-02T00:08:59.551Z [INFO] node-0: snapshot complete up to: index=3
2026-07-02T00:09:02.553Z [ERROR] node-2: failed to take snapshot: error="leadership transfer in progress"
2026-07-02T00:09:07.312Z [INFO] node-0: starting snapshot up to: index=3
2026-07-02T00:09:07.312Z [INFO] snapshot: creating new snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950947312.tmp
2026-07-02T00:09:07.312Z [INFO] snapshot: reaping snapshot: path=/var/lib/raft-node/snapshots/2-3-1782950923792
2026-07-02T00:09:07.312Z [INFO] node-0: no logs to truncate
2026-07-02T00:09:07.312Z [INFO] node-0: snapshot complete up to: index=3
2026-07-02T00:09:07.630Z [ERROR] node-2: failed to take snapshot: error="leadership transfer in progress"
2026-07-02T00:09:10.859Z [ERROR] eventually-workload-commit-progress: commit-progress give-up; cluster healthy but no fresh entry committed: attempts_taken=24 baseline_applied_index=147 last_err="apply rejected: leadership transfer in progress" per_node="map[raft-node-0:8400:map[applied_index:138 last_index:143 leader_id:node-2 reachable:true state:Follower term:169] raft-node-1:8400:map[applied_index:138 last_index:147 leader_id:node-2 reachable:true state:Follower term:169] raft-node-2:8400:map[applied_index:147 last_index:147 leader_id:node-2 reachable:true state:Leader term:169]]"
```
## Root Cause
The root cause is a deadlock during a leadership transfer operation that does not immediately signal any warnings but bites you way into the future by getting the cluster stuck.
- When you initiate a leadership transfer from A to B, the current leader A first tries to get B's logs up to date via an async replication thread.
- The leadership transfer logic is a separate goroutine that waits for this replication before telling B to takeover.
- While all this is happening, A might have to step-down as leader for unrelated reasons (e.g., the lease timer runs out, or it cannot reach a quorum due to network) and say another node C beomes leader.
- When A steps-down as leader, it aborts all replication threads (including the A->B catchup) but the leadership transfer goroutine is still blocked waiting for it to complete (an implementation bug).
- There is no problem so far, because C is the new leader and it does its job for a while.
- Unfortunately, the leadership transfer goroutine in A has set a global flag "leadership transfer in progress" and this flag cannot be unset until it is unblocked.... but it never will be!!!
- Crucially, this flag does not prevent it from participating in elections and the flag does not get reset if it wins a future election.
- So in the future, A can legitimately get relected as leader but it will refuse to accept any client requests and stall on future commits with the reason "leadership transfer in progress".
- When the network is healthy and A is a leader, there is no way to get out of this other than rebooting the node A to clear the flag.
### Source code
When the leadership transfer is interrupted with the replication torn down, the transfer goroutine is stuck forever (deadlocked) here:
https://github.com/hashicorp/raft/blob/4c8f61ac9255bb95fb3b8319dfcf0ae53ab325b6/raft.go#L1003
The leadership transfer flag is only reset here, but this `defer` never runs because nothing is sent on `doneCh`:
https://github.com/hashicorp/raft/blob/4c8f61ac9255bb95fb3b8319dfcf0ae53ab325b6/raft.go#L727
Because of this, when the node gets re-elected as leader in the future, it will always error out, for example:
https://github.com/hashicorp/raft/blob/4c8f61ac9255bb95fb3b8319dfcf0ae53ab325b6/raft.go#L908-L910
## Related Issues
I believe this is different from #498 since the deadlock occurs at a different program location.
Our testing has also revealed safety-property violations which I have reported in #695.
Contributor guide
Research direction
Start in raft.go around lines 1003, 727, and 908-910. Trace how an interrupted leadership transfer waits for replication completion and how the leadership-transfer flag is reset. Done means a step-down cannot leave the transfer goroutine blocked or the flag set, and a later leader can accept client requests without rebooting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100