apache / apache/fluss

Introduce jitter for auto create partition

Open
#775 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/alibaba/fluss/issues) and found nothing similar.

### Motivation

Now, our auto create partition machanism will create may partitions at one point time which may bring mich pressure at one time. We should introduce a jitter so that different tables may create table at different time point to reduce the presure.

Now, 12 tables, 512 bucket per partition, at around 0:10, Fluss with create them, then the isr changes quickly, in bytes decrease, and metrics will looks like:

![Image](https://github.com/user-attachments/assets/0830a840-58d2-4dc1-972e-cf1b83f2aba8)

### Solution
We can introduce jitter to alleviate it first.. But we still need to figure out why it cause isr changes quickly.

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the auto-create partition mechanism and the scheduling or trigger path that creates partitions for multiple tables. Trace how creation times are selected and how partition creation affects ISR and metrics; done means introducing a jitter that spreads table creation over time without breaking partition creation behavior, with coverage for the timing logic.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.