airbnb / airbnb/streamalert

Classifier Lambda not able to recognize kinesis aggregated logs

Open
#1,111 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.9k
Forks
324
PR merge metrics
No merged PRs in 30d

Description

## Background

kinesis aggregation : https://github.com/awslabs/amazon-kinesis-producer/blob/master/aggregation-format.md

used to efficient puts into kinesis data stream to avoid throttling

### Description

classifier lambda exiting with errors like invalid JSON , after careful observation found the problem with aggregated records from kinesis.

KPL uses Google protocol buffers (protobuf) to create a binary file format for this.
NOTE: The Amazon Kinesis Client Library (KCL) implements deaggregation based on this format on the consumer side.

base64 decryption went wrong for classifier because of protobuf

### Steps to Reproduce

feed classifier with KPL generated logs

## Desired Change

de-aggregation solution:
https://github.com/awslabs/kinesis-aggregation/tree/master/python

pip install aws_kinesis_agg can be used to de-aggregate

High level overview of the desired change or outcome.

Contributor guide

Open the contributing guide

Research direction

Start at the classifier Lambda's handling of Kinesis records and compare it with the Python aws_kinesis_agg deaggregation solution linked in the issue. Reproduce the failure with KPL-generated aggregated logs; done means aggregated records are de-aggregated before JSON parsing and no longer produce invalid JSON errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.