Classifier Lambda not able to recognize kinesis aggregated logs
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 324
- PR merge metrics
- No merged PRs in 30d
Description
## Background
kinesis aggregation : https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md
used to efficient puts into kinesis data stream to avoid throttling
### Description
classifier lambda exiting with errors like invalid JSON , after careful observation found the problem with aggregated records from kinesis.
KPL uses Google protocol buffers (protobuf) to create a binary file format for this.
NOTE: The Amazon Kinesis Client Library (KCL) implements deaggregation based on this format on the consumer side.
base64 decryption went wrong for classifier because of protobuf
### Steps to Reproduce
feed classifier with KPL generated logs
## Desired Change
de-aggregation solution:
https://github.com/awslabs/kinesis-aggregation/tree/master/python
pip install aws_kinesis_agg can be used to de-aggregate
High level overview of the desired change or outcome.
Contributor guide
Research direction
Start at the classifier Lambda's handling of Kinesis records and compare it with the Python aws_kinesis_agg deaggregation solution linked in the issue. Reproduce the failure with KPL-generated aggregated logs; done means aggregated records are de-aggregated before JSON parsing and no longer produce invalid JSON errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100