apache / apache/druid

Checking maxRowsPerSegment after segment merge in dynamic partitioning

Open
#11,252 0 comments 0 reactions 0 assignees View on GitHub
Area - Ingestion Improvement
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Description

In this issue, the segment size refers to the row count in the segment. In dynamic partitioning, the task compares `maxRowsPerSegment` and the current `Sink` size to determine whether the segment hits the max segment size limit and needs to be pushed to deep storage. Here, the `Sink` size is the sum of in-memory segment size and the size of all persisted segments _before_ they are merged. As a result, when the rollup is on, the `Sink` size can be larger than the actual segment size. This can create smaller segments than `maxRowsPerSegment` unnecessarily.

Contributor guide

Open the contributing guide

Research direction

Start by locating the dynamic partitioning logic that compares maxRowsPerSegment with the Sink size, then trace how rollup changes the size after persisted segments are merged. Add or update tests covering the post-merge segment row count; done means segments are not pushed unnecessarily early when rollup is enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.