influxdata / influxdata/kapacitor

reduce loglevel for missing values

Open
#1,817 0 comments 4 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Currently, logs for missing values are logged with severity error. In practice, a massive amount of logs is written and it's not possible to filter non-critical "missing value" logs out. I suggest to reduce the log level for individual tasks to severity warning, so that severity "error" only holds application-specific errors and not rule-specific errors.

Examples for rule-based log entried are:
```
ts=2018-02-27T06:53:29.309+01:00 lvl=error msg="failed to realize reduce context from fields" service=kapacitor task_master=main task=chronograf-v1-e2b2f063-9cc9-40d8-bc39-70b24a303714 node=mean3 err="field \"Page_Faults_persec\" missing from point"
ts=2018-02-27T06:53:29.310+01:00 lvl=error msg="failed to realize reduce context from fields" service=kapacitor task_master=main task=chronograf-v1-697f4d4b-ec4d-4bd6-9e5b-295f67669f8a node=mean3 err="field \"Percent_Disk_Time\" missing from point"
ts=2018-02-27T06:53:29.309+01:00 lvl=error msg="error evaluating expression" service=kapacitor task_master=main task=chronograf-v1-142ac251-1b87-4fcf-bf17-ab9b50df533b node=eval2 err="missing value: \"Percent_Free_Space\""
```

Of course, avoiding these errors would be good, but todays Chronograf/Telegraf default settings produce them, so this is a very practical topic. See also https://github.com/influxdata/chronograf/issues/2860

Contributor guide

Open the contributing guide

Research direction

Start by tracing the code paths that emit the messages "failed to realize reduce context from fields" and "error evaluating expression". Check how rule-specific missing-value errors are classified, then verify that these cases use warning severity while application-specific errors remain at error severity.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.