New segments for consuming Kinesis Data Streams not being created
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
When segments reach a threshold row (not the one I specified) the Pinot controller does not create new segments to continue consuming. It does create new ones, once the threshold time is reached.
I have specified the following configurations to the table
"realtime.segment.flush.threshold.rows": "0",
"realtime.segment.flush.threshold.time": "1h",
"realtime.segment.flush.threshold.size": "200M"
However, each segment in S3 is less than 10 megabytes, the segments are closed before reaching the threshold time and size. It seems it gets a default size of 150000 rows and then it does not create new ones up until they get flushed out after an hour.
Contributor guide
Research direction
No files or tests are named. Start by tracing Pinot's realtime Kinesis segment-consumption and flush-threshold handling, then compare the configured row, time, and size thresholds with the observed 150000-row behavior. Done means new segments continue to be created according to the configured thresholds instead of waiting for the one-hour flush.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100