[classifier] optimize sending messages to SQS and Firehose from classifier
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 324
- PR merge metrics
- No merged PRs in 30d
Description
## Background
The vast majority of execution time for the classifier is spent sending record either to SQS or Firehose. Looking at one of the log streams, I can see it takes 1min 10sec to send 33,734 logs to SQS. While this isn't terrible for this many records, we should still investigate how to improve it. Firehose supports larger batches, so it's less of an offender at 16seconds to send the same records.
## Desired Change (options)
- [ ] Implement multiprocessing for the batching of SQS. This could also be problematic, because we will hit service limits and be throttled. Theoretically, multiprocessing could result in worse performance than we already get now.
- [ ] ...
Contributor guide
Research direction
Start by tracing the classifier's paths for sending records to SQS and Firehose, then measure the current batching and send times described in the issue. Evaluate the proposed batching or multiprocessing options against AWS service limits; done means improving throughput without making throttling or overall performance worse.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, cloud
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100