[Feature] Introduce BufferWriteManager in rocksdb for controllable memory consume
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.
### Motivation
Currently, we will have mutiple rocksdb across tablet servers. And every rocksdbs have their own memeory consumtion. It'll be hard for us to control memory usage in sever level.
### Solution
We can just follow Flink's solution to introduce a buffer writer manger to make all rocksdbs in the server share the buffer writer manger.
See
https://issues.apache.org/jira/browse/FLINK-29928
https://issues.apache.org/jira/browse/FLINK-7289
### Anything else?
_No response_
### Willingness to contribute
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing how the repository creates and configures its RocksDB instances, then read the linked Flink designs for BufferWriterManager. The work is complete when multiple RocksDB instances on one tablet server share a controllable server-level memory budget, with tests demonstrating the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100