influxdata / influxdata/kapacitor
reduce loglevel for missing values
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Currently, logs for missing values are logged with severity error. In practice, a massive amount of logs is written and it's not possible to filter non-critical "missing value" logs out. I suggest to reduce the log level for individual tasks to severity warning, so that severity "error" only holds application-specific errors and not rule-specific errors.
Examples for rule-based log entried are:
```
ts=2018-02-27T06:53:29.309+01:00 lvl=error msg="failed to realize reduce context from fields" service=kapacitor task_master=main task=chronograf-v1-e2b2f063-9cc9-40d8-bc39-70b24a303714 node=mean3 err="field \"Page_Faults_persec\" missing from point"
ts=2018-02-27T06:53:29.310+01:00 lvl=error msg="failed to realize reduce context from fields" service=kapacitor task_master=main task=chronograf-v1-697f4d4b-ec4d-4bd6-9e5b-295f67669f8a node=mean3 err="field \"Percent_Disk_Time\" missing from point"
ts=2018-02-27T06:53:29.309+01:00 lvl=error msg="error evaluating expression" service=kapacitor task_master=main task=chronograf-v1-142ac251-1b87-4fcf-bf17-ab9b50df533b node=eval2 err="missing value: \"Percent_Free_Space\""
```
Of course, avoiding these errors would be good, but todays Chronograf/Telegraf default settings produce them, so this is a very practical topic. See also https://github.com/influxdata/chronograf/issues/2860
Contributor guide
Research direction
Start by tracing the code paths that emit the messages "failed to realize reduce context from fields" and "error evaluating expression". Check how rule-specific missing-value errors are classified, then verify that these cases use warning severity while application-specific errors remain at error severity.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100