Repeated empty segments commits for Kinesis merged Shards
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
**Problem**
When shards are merged or reduced in Kinesis, Servers send segment consumed message to Controller with reason: endOfPartition reached.
Controller during commit process can incorrectly determine partition has not reached its end and creates new consuming segment. The servers immediately commit that new segment. This process keeps on repeating.
Error log:
```
2025/11/11 18:41:00.041 WARN [KinesisStreamMetadataProvider] [grizzly-http-server-0] Reached max attempts to check if end of shard reached from checkpoint {"shardId-000000000000":"<>"}. Assuming we have not consumed till end of shard.
```
Code to be re-visited:
https://github.com/apache/pinot/blob/844d4ab8f9fcdc8fa71ffd750ac31ee858a815d0/pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisStreamMetadataProvider.java#L228-L257
Contributor guide
Research direction
Start in pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisStreamMetadataProvider.java at lines 228-257, and trace how the endOfPartition reason and checkpoint are handled during commits. Use the reported repeated empty-segment log as the failure signal; done means merged Kinesis shards no longer cause repeated new segments to be immediately committed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100