apache / apache/pinot

Repeated empty segments commits for Kinesis merged Shards

Open
#17,209 3 comments 0 reactions 0 assignees View on GitHub
bug ingestion kinesis real-time
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

**Problem**
When shards are merged or reduced in Kinesis, Servers send segment consumed message to Controller with reason: endOfPartition reached.
Controller during commit process can incorrectly determine partition has not reached its end and creates new consuming segment. The servers immediately commit that new segment. This process keeps on repeating.

Error log:
```
2025/11/11 18:41:00.041 WARN [KinesisStreamMetadataProvider] [grizzly-http-server-0] Reached max attempts to check if end of shard reached from checkpoint {"shardId-000000000000":"<>"}. Assuming we have not consumed till end of shard.
```

Code to be re-visited:
https://github.com/apache/pinot/blob/844d4ab8f9fcdc8fa71ffd750ac31ee858a815d0/pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisStreamMetadataProvider.java#L228-L257

Contributor guide

Open the contributing guide

Research direction

Start in pinot-plugins/pinot-stream-ingestion/pinot-kinesis/src/main/java/org/apache/pinot/plugin/stream/kinesis/KinesisStreamMetadataProvider.java at lines 228-257, and trace how the endOfPartition reason and checkpoint are handled during commits. Use the reported repeated empty-segment log as the failure signal; done means merged Kinesis shards no longer cause repeated new segments to be immediately committed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.