apache / apache/fluss

[Feature] Control the background job thread nums of rocksdb

Open
#70 0 comments 0 reactions 0 assignees View on GitHub
component=server
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.

### Motivation

Currently, we will create two background threads per bucket to do fluss/compaction which may cause controllable threads.

### Solution

We can control the background threads in server level. Thanksfully, Rocksdb allows us share a threads pool across rocksdbs.
See more in https://github.com/facebook/rocksdb/blob/3ef909248736eb80e7c6b3c7a19753c4dbe9d0c5/include/rocksdb/env.h#L465

### Anything else?

_No response_

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing where the server creates RocksDB instances and where the two background threads per bucket are configured. Read the linked RocksDB Env thread-pool documentation, then determine how a server-level shared pool can control those threads; done means the background thread count is configurable without creating a separate pool per bucket.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.