apache / apache/pinot

Support consumer de-aggregation in Kinesis

Open
#10,152 0 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted kinesis
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Kinesis supports producing aggregated (aka batched) record. Thus, kinesis consumer also has support for de-aggregating the records. Refer - https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-consumer-deaggregation.html

Kinesis provides an `AggregatorUtil` (https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client/src/main/java/software/amazon/kinesis/retrieval/AggregatorUtil.java) that can be used in the `KinesisConsumer` implementation. An example usage of this util can be found in the beam repo (https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/SimplifiedKinesisClient.java#L269)

Even though the records in the batch have the same sequence number, we can append the sub-sequence number to Pinot kinesis' `StreamMessageOffset`. Changes should be fairly trivial to do this.

Labels: `enhancement` , `kinesis`, `help-wanted`

Contributor guide

Open the contributing guide

Research direction

Start by locating the KinesisConsumer implementation and its StreamMessageOffset handling. Review AWS's AggregatorUtil and the Apache Beam SimplifiedKinesisClient example linked in the issue, then verify that aggregated records are de-aggregated and each sub-record receives an offset containing its sub-sequence number.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.