Improvement: Multiprocessing of classification
Open
classifier
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 324
- PR merge metrics
- No merged PRs in 30d
Description
## Background
AWS Lambda supports a [thread limit](http://docs.aws.amazon.com/lambda/latest/dg/limits.html) of 1024 per execution. We currently are not taking advantage of this feature when processing logs with StreamAlert.
## Desired Outcome
Usage of Python's [multiprocessing](https://docs.python.org/2/library/multiprocessing.html) builtin library to parallelize the classification and rules matching engine.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by locating the classification and rules-matching engine, then review Python multiprocessing and AWS Lambda's thread limit; done means classification and rule matching are parallelized during log processing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, cloud, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100