apache / apache/fluss

[Feature] Introduce BufferWriteManager in rocksdb for controllable memory consume

Open
#69 2 comments 1 reaction 0 assignees View on GitHub
component=server
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.

### Motivation

Currently, we will have mutiple rocksdb across tablet servers. And every rocksdbs have their own memeory consumtion. It'll be hard for us to control memory usage in sever level.

### Solution

We can just follow Flink's solution to introduce a buffer writer manger to make all rocksdbs in the server share the buffer writer manger.
See
https://issues.apache.org/jira/browse/FLINK-29928
https://issues.apache.org/jira/browse/FLINK-7289

### Anything else?

_No response_

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing how the repository creates and configures its RocksDB instances, then read the linked Flink designs for BufferWriterManager. The work is complete when multiple RocksDB instances on one tablet server share a controllable server-level memory budget, with tests demonstrating the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.