awslabs / awslabs/kinesis-aggregation
Is data loss possible when consume aggregated records by AWS lambda?
- Dominant language
- Java
- Stars
- 379
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
I've read the following warning from [this page ](https://www.npmjs.com/package/aws-kinesis-agg)
**Caution - this module is only suitable for low-value messages which are processed in aggregate. Do not use Kinesis Aggregation for data which is sensitive or where every message must be delivered, and where the KCL (including with AWS Lambda) is used for processing. [DATA LOSS CAN OCCUR.](https://github.com/awslabs/kinesis-aggregation/blob/potential_data_loss.md)**
But as I know, records are not consumed by KCL in lambda and need to be deaggregated manually by invoking `deaggregate`, Is it still possible data loss in this case?
Contributor guide
Research direction
Read the linked potential_data_loss.md warning first, then trace how AWS Lambda and KCL deliver aggregated Kinesis records before manual deaggregation with deaggregate. Confirm whether this consumption path can lose records and document the conclusion with the relevant conditions and evidence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, node.js
- Domain
- cloud, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100