[server] Drop table (partition) may fail when tablet server crashs
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
main (development)
### Please describe the bug 🐞
When we drop a table or partition, the corresponding buckets should be deleted. However, if a tablet server crashes and restarts during dropping the table or partition, the replica in the crashed tablet server of buckets can not successfully removed and keep residual data. Currently, if we restart coordinator server, the residual bucket can be deleted.
I think it is because the crashed tablet server could not receive the StopReplicaRequest and thus do not send a response (`DeleteReplicaResponseReceivedEvent`). There is neither a retry mechanism or an independent thread to periodically check if there are buckets should be deleted.
### Solution
Add a manager to periodically check the difference between coordinatorContext and Zookeeper nodes
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing StopReplicaRequest and DeleteReplicaResponseReceivedEvent, then inspect how coordinatorContext is compared with the ZooKeeper nodes during table or partition deletion. The fix is complete when a tablet server crash and restart no longer leaves residual bucket data, without requiring a coordinator restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100