Create a custom metric to track number of failing rules
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 324
- PR merge metrics
- No merged PRs in 30d
Description
## Background
Right now, we [log an exception](https://github.com/airbnb/streamalert/blob/f336c7850235096052c36aa8836f577a506c02c3/stream_alert/shared/rule.py#L160-L161) when a rule fails, but do not track this via any reportable means.
## Desired Change
Create a custom metric that gets logged when a rule causes and exception. There is no need to get granular (as far as reporting "what rule caused the exception").. we just need something we can create an alarm on then triage later to determine the the offending rules.
Contributor guide
Research direction
Start in stream_alert/shared/rule.py at the exception logging referenced around lines 160-161, and trace how rule failures are handled. Confirm where the custom metric should be emitted and how existing reporting or monitoring hooks work. Done means each rule exception produces a reportable metric that can be used to create an alarm without identifying the individual rule.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100