Determine Number of Rows for a Realtime Segment Based on a Config
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
At present we have two types of configs:
1. Set threshold based on size
2. Set threshold based on number of rows
Regarding the second type, while the value set in those configs is proportional to the number of rows in a segment, it's not the exact value. It seems to divide the number of rows by the maximum number of kafka partitions in a server.
Feature: It would be great to have a config that determines the number of rows in a segment exactly. If the config says 1M, then the segments should be flushed after 1M rows.
cc: @wirybeaver
Contributor guide
Research direction
Start by tracing the existing size-based and row-count segment threshold configuration and the realtime segment flush logic. Check how the maximum Kafka partition count affects the current row threshold; done means a configured value such as 1M causes a segment to flush after exactly 1M rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- database
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100