Revisit Kinesis offset handling for inclusive start sequence numbers and empty batches
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
### Motivation
`pinot-plugins/pinot-stream-ingestion/pinot-kinesis` still has TODOs in:
- `KinesisConsumer#getKinesisMessageBatch()`
- `KinesisStreamMetadataProvider#computePartitionGroupMetadata(...)`
Both comments call out a potential first-message loss because Pinot starts from an inclusive sequence number but requests `AFTER_SEQUENCE_NUMBER`. The consumer also notes that an empty batch may be treated as fully caught up even if Kinesis can still have records available.
### Scope
- Reproduce the first-message-loss edge case for new shards and partitions.
- Clarify Kinesis empty-batch semantics and Pinot's caught-up behavior.
- Fix offset handling if needed and add regression coverage.
### Notes
Observed on `upstream/master` on April 15, 2026.
Contributor guide
Research direction
Start by reading the TODOs in pinot-plugins/pinot-stream-ingestion/pinot-kinesis, especially KinesisConsumer#getKinesisMessageBatch() and KinesisStreamMetadataProvider#computePartitionGroupMetadata(...). Reproduce the new-shard and new-partition cases, then verify Kinesis empty-batch semantics and Pinot’s caught-up behavior. Done means offset handling is correct and regression coverage demonstrates that the first message is not lost.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100