airbnb / airbnb/streamalert

Improvement: Support JSON key wildcarding

Open
#150 3 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Python
Stars
2.9k
Forks
324
PR merge metrics
No merged PRs in 30d

Description

**Problem**

Some JSON logs have keys that are dynamic

Example: CarbonBlack's `feed.storage.hit.process`
- Example key: alliance_data_

Where is `bit9endpointvisibility`, `bit9earlyaccess`, or one of a dozen other feeds. These feed names can grow and change over time.

**Proposal**

Support wildcarding, like so:

```
"alliance_data_*": "string",
"alliance_link_*": "string",
"alliance_score_*": "string",
"alliance_updated_*": "string"
```

This would need to be implemented at the schema level, meaning it should work if they keys are specified under `schema`, `optional_top_level_keys` or `envelope_keys`

Contributor guide

Open the contributing guide

Research direction

Start by tracing the schema handling for JSON keys, including schema, optional_top_level_keys, and envelope_keys. Determine how wildcard keys such as alliance_data_* should be matched across each location, then verify that dynamic feed names are accepted without changing explicit key behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.