influxdata / influxdata/influxdb

"failed to store statistics" messages log level is "info" - should be "error"

Open
#17,486 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Proposal:__
"failed to store statistics" message has log level "info". This is a critical error because data is being lost.
```
Mar 29 12:49:10 pi4 influxd[6193]: ts=2020-03-29T09:49:10.363211Z **lvl=info** msg="failed to store statistics" log_id=0LqPTBQ0000 service=monitor error=timeout
Mar 29 13:01:29 pi4 influxd[11256]: ts=2020-03-29T10:01:29.466153Z **lvl=info** msg="Write failed" log_id=0LqReCk0000 service=write shard=267 error="store is closed"
Mar 29 13:01:29 pi4 influxd[11256]: ts=2020-03-29T10:01:29.562252Z **lvl=info** msg="Write failed" log_id=0LqReCk0000 service=write shard=267 error="store is closed"
Mar 29 13:01:29 pi4 influxd[11256]: ts=2020-03-29T10:01:29.763595Z **lvl=info** msg="Write failed" log_id=0LqReCk0000 service=write shard=267 error="store is closed"
Mar 29 13:21:40 pi4 influxd[12856]: ts=2020-03-29T10:21:40.387157Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:21:50 pi4 influxd[12856]: ts=2020-03-29T10:21:50.764947Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:22:01 pi4 influxd[12856]: ts=2020-03-29T10:22:01.152171Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:22:11 pi4 influxd[12856]: ts=2020-03-29T10:22:11.514534Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:22:21 pi4 influxd[12856]: ts=2020-03-29T10:22:21.889965Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:22:32 pi4 influxd[12856]: ts=2020-03-29T10:22:32.271676Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:22:42 pi4 influxd[12856]: ts=2020-03-29T10:22:42.617266Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:22:52 pi4 influxd[12856]: ts=2020-03-29T10:22:52.970765Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:23:03 pi4 influxd[12856]: ts=2020-03-29T10:23:03.337678Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:23:13 pi4 influxd[12856]: ts=2020-03-29T10:23:13.709623Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
```
My configuration is: rpi4 4gb ram + openhab2 + influxdb used as persistence service.
openhab2 throws the following errors:
```
2020-03-29 13:31:43.573 [ERROR] [org.influxdb.impl.BatchProcessor ] - Batch could not be sent. Data will be lost
java.lang.RuntimeException: {"error":"timeout"}
2020-03-29 13:32:23.572 [ERROR] [org.influxdb.impl.BatchProcessor ] - Batch could not be sent. Data will be lost
java.lang.RuntimeException: {"error":"timeout"}
2020-03-29 13:33:03.570 [ERROR] [org.influxdb.impl.BatchProcessor ] - Batch could not be sent. Data will be lost
java.lang.RuntimeException: {"error":"timeout"}
2020-03-29 13:33:13.587 [ERROR] [org.influxdb.impl.BatchProcessor ] - Batch could not be sent. Data will be lost
java.lang.RuntimeException: {"error":"timeout"}
2020-03-29 13:34:13.570 [ERROR] [org.influxdb.impl.BatchProcessor ] - Batch could not be sent. Data will be lost
java.lang.RuntimeException: {"error":"timeout"}
2020-03-29 13:35:13.581 [ERROR] [org.influxdb.impl.BatchProcessor ] - Batch could not be sent. Data will be lost
java.lang.RuntimeException: {"error":"timeout"}
```

__Current behavior:__
"failed to store statistics" messages are stored in influxdb.log only if logging.level="info"

__Desired behavior:__
"failed to store statistics" messages to be tagged with "error" log level. Reasons are explained above.

__Alternatives considered:__
as a workaround I keep logging.level="info" but this setting is bloating the log file unnecessarily.

__Use case:__
Why is this important (helps with prioritizing requests)?
Because "failed to store statistics" messages are usually critical errors from the workflow perspective, if the lost data is critical within the context of the 3rd party apps that write data to influxdb. In my case, I persistently lose important telemetry data and influxdb marks these issues as "info"...

Contributor guide

Open the contributing guide

Research direction

The issue names no source file or test. Start by locating the monitor entry point that emits the "failed to store statistics" message, then inspect how its log level is selected. Done means the message is emitted at error level and the relevant existing tests, if present, verify that behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.