airbnb / airbnb/streamalert

Create a custom metric to track number of failing rules

Open
#985 0 comments 0 reactions 0 assignees View on GitHub
metrics rules rules engine
Dominant language
Python
Stars
2.9k
Forks
324
PR merge metrics
No merged PRs in 30d

Description

## Background

Right now, we [log an exception](https://github.com/airbnb/streamalert/blob/f336c7850235096052c36aa8836f577a506c02c3/stream_alert/shared/rule.py#L160-L161) when a rule fails, but do not track this via any reportable means.

## Desired Change

Create a custom metric that gets logged when a rule causes and exception. There is no need to get granular (as far as reporting "what rule caused the exception").. we just need something we can create an alarm on then triage later to determine the the offending rules.

Contributor guide

Open the contributing guide

Research direction

Start in stream_alert/shared/rule.py at the exception logging referenced around lines 160-161, and trace how rule failures are handled. Confirm where the custom metric should be emitted and how existing reporting or monitoring hooks work. Done means each rule exception produces a reportable metric that can be used to create an alarm without identifying the individual rule.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.