apache / apache/pinot

Determine Number of Rows for a Realtime Segment Based on a Config

Open
#10,137 13 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

At present we have two types of configs:

1. Set threshold based on size
2. Set threshold based on number of rows

Regarding the second type, while the value set in those configs is proportional to the number of rows in a segment, it's not the exact value. It seems to divide the number of rows by the maximum number of kafka partitions in a server.

Feature: It would be great to have a config that determines the number of rows in a segment exactly. If the config says 1M, then the segments should be flushed after 1M rows.

cc: @wirybeaver

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing size-based and row-count segment threshold configuration and the realtime segment flush logic. Check how the maximum Kafka partition count affects the current row threshold; done means a configured value such as 1M causes a segment to flush after exactly 1M rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.