tikv / tikv/pd

[Bug] Failover Not Failsafe - Data Loss Possible

Open
#10,788 1 comment 0 reactions 0 assignees View on GitHub
contribution needs-type
Dominant language
Go
Stars
1.2k
Forks
783
Avg merge
5d 21h
Merged PRs (30d)
36

Description

## Description
Failover process is not failsafe. Data loss is possible during failover scenarios. Leader failure recovery doesn't guarantee consistency. Some write operations may not be replicated before leader failure.

## Steps to Reproduce
1. TiKV leader processes write operation
2. Write not yet replicated to followers
3. Leader crashes suddenly
4. Failover elects new leader
5. Unquorum writes are lost

## Environment Information
- Replication: Multi-raft based
- Failover: Leader election
- Consistency: Strong (intended)
- Recovery: Incomplete

## Expected Behavior
- No data loss during failover
- Write consistency guaranteed
- All committed writes persisted
- Leader handles unconfirmed writes safely

## Actual Behavior
- Data loss possible
- Unquorum writes not replicated
- Failover causes inconsistency
- No durability guarantee

## Code Reference
- File: `raftstore/store.rs`
- Method: `handle_leader_failure()` - incomplete
- Missing: Write durability check
- Missing: Quorum confirmation

## Additional Context
Level 1 reliability issue. Fix requires:
- Ensuring write quorum before confirmation
- Handling unconfirmed writes on leader failure
- Implementing write durability checks
- Failover safety testing

## Suggested Labels
`reliability`, `data-loss`, `failover`, `consistency`, `critical`

Contributor guide

Open the contributing guide

Research direction

Start with raftstore/store.rs and the handle_leader_failure() method referenced in the issue. Trace the failover and write-confirmation behavior, then add failover safety tests covering unreplicated and committed writes; done means committed writes remain durable and consistency is preserved after leader failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.