clearup useless fluss partition info in writer metadata cache
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
0.9.0 (latest release)
### Please describe the bug 🐞
Currently, if a partition written job run for a long time, fluss partition info will increase more and more, and never be cleanup. Most of the partition info is no longer used. This will occupy a lot cpu usage for `Sender#sendWriteData ->Sender#sendWriteData -> writeBatches.entrySet()` and `RecordAccumulator#drain - RecordAccumulator#drainBatchesForOneNode -> Cluster#getAvailableBucketsForPhysicalTablePath`
Only restart the job, need to clearup:
* useless PhysicalTablePath in org.apache.fluss.client.write.RecordAccumulator#writeBatches
* uneless parittion info in Cluster
### Solution
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with org.apache.fluss.client.write.RecordAccumulator#writeBatches and the Sender#sendWriteData and RecordAccumulator#drain call paths, then inspect Cluster#getAvailableBucketsForPhysicalTablePath. Reproduce a long-running partition-writing job and observe whether obsolete PhysicalTablePath and partition metadata continue accumulating. Done means unused entries are cleared without affecting active writes and the reported CPU-heavy iterations no longer grow with job duration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100