airbnb / airbnb/streamalert

Performance: Optimize schema matching

Open
#249 2 comments 0 reactions 0 assignees View on GitHub
improvement log parsers performance question
Dominant language
Python
Stars
2.9k
Forks
324
PR merge metrics
No merged PRs in 30d

Description

**What**

Schema checks are performed in the same order against each log due to it being an OrderedDict. This can be problematic if your highest volume log types (ex: osquery) happen to have their schema at the end of the OrderedDict.

This issue was identified when we had a sizeable log type not match any schemas, causing the processor to loop through and try each schema. We then reasoned about how this is a larger problem.

**Potential Solutions**

1. Don't use an OrderedDict for JSON schemas and/or randomize schema ordering to smooth out the time it takes to match each log to a schema

2. Document/communicate how users should order their schemas according to volume

(comment by @ryandeivert): This is actually probably the best approach here. This would be especially helpful for services that have a wide variety of schemas (carbonblack), giving preference to the most common logs first.

3. Somehow better inform the processor what log type(s) it's going to process. Example: most SIEM's have you define what index or category a given log is going to go to. We might not be able to achieve the same thing due to how Kinesis works, but just putting this data point here as a reference

~4. Classification Lambda, which then sends to Rule Processor Lambda~
* The above is implemented.

5. Some other better idea (likely the solution)

Contributor guide

Open the contributing guide

Research direction

Start by tracing the processor's schema-matching loop and reviewing the implemented Classification Lambda noted in the issue. Compare the proposed ordering, documentation, and routing directions with current behavior, then establish an agreed approach and measure whether schema matching improves for high-volume and unmatched log types.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, json, python
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.