[Bug] Failover Not Failsafe - Data Loss Possible
- Dominant language
- Go
- Stars
- 1.2k
- Forks
- 783
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 36
Description
## Description
Failover process is not failsafe. Data loss is possible during failover scenarios. Leader failure recovery doesn't guarantee consistency. Some write operations may not be replicated before leader failure.
## Steps to Reproduce
1. TiKV leader processes write operation
2. Write not yet replicated to followers
3. Leader crashes suddenly
4. Failover elects new leader
5. Unquorum writes are lost
## Environment Information
- Replication: Multi-raft based
- Failover: Leader election
- Consistency: Strong (intended)
- Recovery: Incomplete
## Expected Behavior
- No data loss during failover
- Write consistency guaranteed
- All committed writes persisted
- Leader handles unconfirmed writes safely
## Actual Behavior
- Data loss possible
- Unquorum writes not replicated
- Failover causes inconsistency
- No durability guarantee
## Code Reference
- File: `raftstore/store.rs`
- Method: `handle_leader_failure()` - incomplete
- Missing: Write durability check
- Missing: Quorum confirmation
## Additional Context
Level 1 reliability issue. Fix requires:
- Ensuring write quorum before confirmation
- Handling unconfirmed writes on leader failure
- Implementing write durability checks
- Failover safety testing
## Suggested Labels
`reliability`, `data-loss`, `failover`, `consistency`, `critical`
Contributor guide
Research direction
Start with raftstore/store.rs and the handle_leader_failure() method referenced in the issue. Trace the failover and write-confirmation behavior, then add failover safety tests covering unreplicated and committed writes; done means committed writes remain durable and consistency is preserved after leader failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100