apache / apache/fluss

clearup useless fluss partition info in writer metadata cache

Open
#3,362 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Fluss version

0.9.0 (latest release)

### Please describe the bug 🐞

Currently, if a partition written job run for a long time, fluss partition info will increase more and more, and never be cleanup. Most of the partition info is no longer used. This will occupy a lot cpu usage for `Sender#sendWriteData ->Sender#sendWriteData -> writeBatches.entrySet()` and `RecordAccumulator#drain - RecordAccumulator#drainBatchesForOneNode -> Cluster#getAvailableBucketsForPhysicalTablePath`

Only restart the job, need to clearup:
* useless PhysicalTablePath in org.apache.fluss.client.write.RecordAccumulator#writeBatches
* uneless parittion info in Cluster

### Solution

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with org.apache.fluss.client.write.RecordAccumulator#writeBatches and the Sender#sendWriteData and RecordAccumulator#drain call paths, then inspect Cluster#getAvailableBucketsForPhysicalTablePath. Reproduce a long-running partition-writing job and observe whether obsolete PhysicalTablePath and partition metadata continue accumulating. Done means unused entries are cleared without affecting active writes and the reported CPU-heavy iterations no longer grow with job duration.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.