airbnb / airbnb/streamalert

[classifier] optimize sending messages to SQS and Firehose from classifier

Open
#869 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Python
Stars
2.9k
Forks
324
PR merge metrics
No merged PRs in 30d

Description

## Background

The vast majority of execution time for the classifier is spent sending record either to SQS or Firehose. Looking at one of the log streams, I can see it takes 1min 10sec to send 33,734 logs to SQS. While this isn't terrible for this many records, we should still investigate how to improve it. Firehose supports larger batches, so it's less of an offender at 16seconds to send the same records.

## Desired Change (options)

- [ ] Implement multiprocessing for the batching of SQS. This could also be problematic, because we will hit service limits and be throttled. Theoretically, multiprocessing could result in worse performance than we already get now.
- [ ] ...

Contributor guide

Open the contributing guide

Research direction

Start by tracing the classifier's paths for sending records to SQS and Firehose, then measure the current batching and send times described in the issue. Evaluate the proposed batching or multiprocessing options against AWS service limits; done means improving throughput without making throttling or overall performance worse.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, cloud
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.