apache / apache/pinot

Revisit Kinesis offset handling for inclusive start sequence numbers and empty batches

Open
#18,220 0 comments 0 reactions 0 assignees View on GitHub
AWS bug ingestion java kinesis priority: medium real-time
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

### Motivation
`pinot-plugins/pinot-stream-ingestion/pinot-kinesis` still has TODOs in:
- `KinesisConsumer#getKinesisMessageBatch()`
- `KinesisStreamMetadataProvider#computePartitionGroupMetadata(...)`

Both comments call out a potential first-message loss because Pinot starts from an inclusive sequence number but requests `AFTER_SEQUENCE_NUMBER`. The consumer also notes that an empty batch may be treated as fully caught up even if Kinesis can still have records available.

### Scope
- Reproduce the first-message-loss edge case for new shards and partitions.
- Clarify Kinesis empty-batch semantics and Pinot's caught-up behavior.
- Fix offset handling if needed and add regression coverage.

### Notes
Observed on `upstream/master` on April 15, 2026.

Contributor guide

Open the contributing guide

Research direction

Start by reading the TODOs in pinot-plugins/pinot-stream-ingestion/pinot-kinesis, especially KinesisConsumer#getKinesisMessageBatch() and KinesisStreamMetadataProvider#computePartitionGroupMetadata(...). Reproduce the new-shard and new-partition cases, then verify Kinesis empty-batch semantics and Pinot’s caught-up behavior. Done means offset handling is correct and regression coverage demonstrates that the first message is not lost.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.