Performance: Optimize schema matching
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 324
- PR merge metrics
- No merged PRs in 30d
Description
**What**
Schema checks are performed in the same order against each log due to it being an OrderedDict. This can be problematic if your highest volume log types (ex: osquery) happen to have their schema at the end of the OrderedDict.
This issue was identified when we had a sizeable log type not match any schemas, causing the processor to loop through and try each schema. We then reasoned about how this is a larger problem.
**Potential Solutions**
1. Don't use an OrderedDict for JSON schemas and/or randomize schema ordering to smooth out the time it takes to match each log to a schema
2. Document/communicate how users should order their schemas according to volume
(comment by @ryandeivert): This is actually probably the best approach here. This would be especially helpful for services that have a wide variety of schemas (carbonblack), giving preference to the most common logs first.
3. Somehow better inform the processor what log type(s) it's going to process. Example: most SIEM's have you define what index or category a given log is going to go to. We might not be able to achieve the same thing due to how Kinesis works, but just putting this data point here as a reference
~4. Classification Lambda, which then sends to Rule Processor Lambda~
* The above is implemented.
5. Some other better idea (likely the solution)
Contributor guide
Research direction
Start by tracing the processor's schema-matching loop and reviewing the implemented Classification Lambda noted in the issue. Compare the proposed ordering, documentation, and routing directions with current behavior, then establish an agreed approach and measure whether schema matching improves for high-volume and unmatched log types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, json, python
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100