awslabs / awslabs/kinesis-aggregation

Is data loss possible when consume aggregated records by AWS lambda?

Open
#175 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
379
Forks
145
PR merge metrics
No merged PRs in 30d

Description

I've read the following warning from [this page ](https://www.npmjs.com/package/aws-kinesis-agg)

**Caution - this module is only suitable for low-value messages which are processed in aggregate. Do not use Kinesis Aggregation for data which is sensitive or where every message must be delivered, and where the KCL (including with AWS Lambda) is used for processing. [DATA LOSS CAN OCCUR.](https://github.com/awslabs/kinesis-aggregation/blob/potential_data_loss.md)**

But as I know, records are not consumed by KCL in lambda and need to be deaggregated manually by invoking `deaggregate`, Is it still possible data loss in this case?

Contributor guide

Open the contributing guide

Research direction

Read the linked potential_data_loss.md warning first, then trace how AWS Lambda and KCL deliver aggregated Kinesis records before manual deaggregation with deaggregate. Confirm whether this consumption path can lose records and document the conclusion with the relevant conditions and evidence.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, node.js
Domain
cloud, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.