apache / apache/fluss

[server] Drop table (partition) may fail when tablet server crashs

Open
#1,629 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Fluss version

main (development)

### Please describe the bug 🐞

When we drop a table or partition, the corresponding buckets should be deleted. However, if a tablet server crashes and restarts during dropping the table or partition, the replica in the crashed tablet server of buckets can not successfully removed and keep residual data. Currently, if we restart coordinator server, the residual bucket can be deleted.

I think it is because the crashed tablet server could not receive the StopReplicaRequest and thus do not send a response (`DeleteReplicaResponseReceivedEvent`). There is neither a retry mechanism or an independent thread to periodically check if there are buckets should be deleted.

### Solution
Add a manager to periodically check the difference between coordinatorContext and Zookeeper nodes
_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing StopReplicaRequest and DeleteReplicaResponseReceivedEvent, then inspect how coordinatorContext is compared with the ZooKeeper nodes during table or partition deletion. The fix is complete when a tablet server crash and restart no longer leaves residual bucket data, without requiring a coordinator restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.