apache / apache/pinot

Realtime to Offline segment conversion: Provision for rolling offline segments based on segment size

Open
#6,989 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

REALTIME tables segment size can be controlled using `realtime.segment.flush.threshold.segment.size` (say "100M")
We don't have any similar configuration for OFFLINE tables which can be used during Realtime to Offline table conversion.

There is a configuration `maxNumRecordsPerSegment` with which we can indirectly achieve desired size. However, having a direct size based control will be really helpful and straight forward.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the realtime-to-offline table conversion and the existing maxNumRecordsPerSegment configuration. Compare that path with the realtime.segment.flush.threshold.segment.size setting, then verify that offline conversion supports a direct segment-size control and that the resulting segments follow the configured limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.