influxdata / influxdata/influxdb
"failed to store statistics" messages log level is "info" - should be "error"
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Proposal:__
"failed to store statistics" message has log level "info". This is a critical error because data is being lost.
```
Mar 29 12:49:10 pi4 influxd[6193]: ts=2020-03-29T09:49:10.363211Z **lvl=info** msg="failed to store statistics" log_id=0LqPTBQ0000 service=monitor error=timeout
Mar 29 13:01:29 pi4 influxd[11256]: ts=2020-03-29T10:01:29.466153Z **lvl=info** msg="Write failed" log_id=0LqReCk0000 service=write shard=267 error="store is closed"
Mar 29 13:01:29 pi4 influxd[11256]: ts=2020-03-29T10:01:29.562252Z **lvl=info** msg="Write failed" log_id=0LqReCk0000 service=write shard=267 error="store is closed"
Mar 29 13:01:29 pi4 influxd[11256]: ts=2020-03-29T10:01:29.763595Z **lvl=info** msg="Write failed" log_id=0LqReCk0000 service=write shard=267 error="store is closed"
Mar 29 13:21:40 pi4 influxd[12856]: ts=2020-03-29T10:21:40.387157Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:21:50 pi4 influxd[12856]: ts=2020-03-29T10:21:50.764947Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:22:01 pi4 influxd[12856]: ts=2020-03-29T10:22:01.152171Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:22:11 pi4 influxd[12856]: ts=2020-03-29T10:22:11.514534Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:22:21 pi4 influxd[12856]: ts=2020-03-29T10:22:21.889965Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:22:32 pi4 influxd[12856]: ts=2020-03-29T10:22:32.271676Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:22:42 pi4 influxd[12856]: ts=2020-03-29T10:22:42.617266Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:22:52 pi4 influxd[12856]: ts=2020-03-29T10:22:52.970765Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:23:03 pi4 influxd[12856]: ts=2020-03-29T10:23:03.337678Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
Mar 29 13:23:13 pi4 influxd[12856]: ts=2020-03-29T10:23:13.709623Z **lvl=info** msg="failed to store statistics" log_id=0LqS6FW0000 service=monitor error=timeout
```
My configuration is: rpi4 4gb ram + openhab2 + influxdb used as persistence service.
openhab2 throws the following errors:
```
2020-03-29 13:31:43.573 [ERROR] [org.influxdb.impl.BatchProcessor ] - Batch could not be sent. Data will be lost
java.lang.RuntimeException: {"error":"timeout"}
2020-03-29 13:32:23.572 [ERROR] [org.influxdb.impl.BatchProcessor ] - Batch could not be sent. Data will be lost
java.lang.RuntimeException: {"error":"timeout"}
2020-03-29 13:33:03.570 [ERROR] [org.influxdb.impl.BatchProcessor ] - Batch could not be sent. Data will be lost
java.lang.RuntimeException: {"error":"timeout"}
2020-03-29 13:33:13.587 [ERROR] [org.influxdb.impl.BatchProcessor ] - Batch could not be sent. Data will be lost
java.lang.RuntimeException: {"error":"timeout"}
2020-03-29 13:34:13.570 [ERROR] [org.influxdb.impl.BatchProcessor ] - Batch could not be sent. Data will be lost
java.lang.RuntimeException: {"error":"timeout"}
2020-03-29 13:35:13.581 [ERROR] [org.influxdb.impl.BatchProcessor ] - Batch could not be sent. Data will be lost
java.lang.RuntimeException: {"error":"timeout"}
```
__Current behavior:__
"failed to store statistics" messages are stored in influxdb.log only if logging.level="info"
__Desired behavior:__
"failed to store statistics" messages to be tagged with "error" log level. Reasons are explained above.
__Alternatives considered:__
as a workaround I keep logging.level="info" but this setting is bloating the log file unnecessarily.
__Use case:__
Why is this important (helps with prioritizing requests)?
Because "failed to store statistics" messages are usually critical errors from the workflow perspective, if the lost data is critical within the context of the 3rd party apps that write data to influxdb. In my case, I persistently lose important telemetry data and influxdb marks these issues as "info"...
Contributor guide
Research direction
The issue names no source file or test. Start by locating the monitor entry point that emits the "failed to store statistics" message, then inspect how its log level is selected. Done means the message is emitted at error level and the relevant existing tests, if present, verify that behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100