influxdata / influxdata/influxdb

Missing Data in Grafana Graphs Despite Continuous Netdata + InfluxDB Collection and Temporary Network Loss

Open
#26,102 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

### **Issue Description:**
The system, running **Debian 10 on an Intel NUC**, was operating normally. However, there was a network connection interruption at approximately 12:10 PM on December 27. Despite the machine remaining powered on, Grafana's graphs stopped updating.

The data does not align correctly with the time axis in Grafana. There is a **29-minute gap** in the displayed data from **12:06 to 12:35**, even though **InfluxDB contains the data for this period**. The issue was resolved only **after rebooting the machine**.

### **Primary Issue:**
Grafana's graphs exhibit **missing segments**, despite the data being properly recorded in InfluxDB and the machine running during that time period.

- ![Image](https://github.com/user-attachments/assets/0a0b84f4-1d44-41c2-8258-b3fd38f962fb)
- ![Image](https://github.com/user-attachments/assets/cb512c55-366e-4226-ab14-9c60d47c76ea)
- ![Image](https://github.com/user-attachments/assets/de393ecb-fc01-4506-a9ec-37e9f6f5c9c5)
- ![Image](https://github.com/user-attachments/assets/49c7e77a-a808-490a-91ce-ff750be46d01)
- ![Image](https://github.com/user-attachments/assets/d735ab39-a290-4a72-9bd2-2394fb893eca)

- Data **is present** in InfluxDB, and the machine **remained powered on**.
- Used InfluxDB shell (`SELECT * FROM measurement`) and confirmed that the data exists for the missing period.
```- root@DEB-NUC11PAH:~# influx
Connected to http://localhost:8086 version 1.8.1
InfluxDB shell version: 1.8.1
> use netdata
Using database netdata
> precision rfc3339
> SELECT * FROM "netdata.system.uptime.uptime"
name: netdata.system.uptime.uptime
time host value
---- ---- -----
2024-12-27T11:59:58Z DEB-NUC11PAH 30207.4713333
2024-12-27T12:00:58Z DEB-NUC11PAH 30267.5
2024-12-27T12:01:58Z DEB-NUC11PAH 30327.5005
2024-12-27T12:02:58Z DEB-NUC11PAH 30387.1833333
2024-12-27T12:03:58Z DEB-NUC11PAH 30447.4155
2024-12-27T12:04:58Z DEB-NUC11PAH 30507.505
2024-12-27T12:05:58Z DEB-NUC11PAH 30567.5081667
2024-12-27T12:06:58Z DEB-NUC11PAH 30627.2443333
2024-12-27T12:07:58Z DEB-NUC11PAH 30687.3618333
2024-12-27T12:08:58Z DEB-NUC11PAH 30747.4713333
2024-12-27T12:09:58Z DEB-NUC11PAH 30807.3278333
2024-12-27T12:10:58Z DEB-NUC11PAH 30867.3898333
2024-12-27T12:11:58Z DEB-NUC11PAH 30927.4505
2024-12-27T12:12:58Z DEB-NUC11PAH 30987.215
2024-12-27T12:13:58Z DEB-NUC11PAH 31047.5
2024-12-27T12:14:58Z DEB-NUC11PAH 31107.4563333
2024-12-27T12:15:58Z DEB-NUC11PAH 31167.3141667
2024-12-27T12:16:58Z DEB-NUC11PAH 31227.5053333
2024-12-27T12:17:58Z DEB-NUC11PAH 31287.3616667
2024-12-27T12:18:58Z DEB-NUC11PAH 31347.2481667
2024-12-27T12:19:58Z DEB-NUC11PAH 31407.5
2024-12-27T12:20:58Z DEB-NUC11PAH 31467.3561667
2024-12-27T12:21:58Z DEB-NUC11PAH 31527.3281667
2024-12-27T12:22:58Z DEB-NUC11PAH 31587.415
2024-12-27T12:23:58Z DEB-NUC11PAH 31647.4481667
2024-12-27T12:24:58Z DEB-NUC11PAH 31707.4476667
2024-12-27T12:25:58Z DEB-NUC11PAH 31767.1603333
2024-12-27T12:26:58Z DEB-NUC11PAH 31827.5005
2024-12-27T12:27:58Z DEB-NUC11PAH 31887.485
2024-12-27T12:28:58Z DEB-NUC11PAH 31947.4573333
2024-12-27T12:29:58Z DEB-NUC11PAH 32007.2991667
2024-12-27T12:30:58Z DEB-NUC11PAH 32067.4161667
2024-12-27T12:31:58Z DEB-NUC11PAH 32127.4495
2024-12-27T12:32:58Z DEB-NUC11PAH 32187.2456667
2024-12-27T12:33:58Z DEB-NUC11PAH 32247.3033333
2024-12-27T12:34:58Z DEB-NUC11PAH 32307.5
2024-12-27T12:35:58Z DEB-NUC11PAH 32367.4711667
2024-12-27T12:36:58Z DEB-NUC11PAH 32427.444
2024-12-27T12:37:58Z DEB-NUC11PAH 32487.2991667
2024-12-27T12:38:58Z DEB-NUC11PAH 32547.4136667
2024-12-27T12:39:58Z DEB-NUC11PAH 32607.5086667
2024-12-27T12:40:58Z DEB-NUC11PAH 32667.2455
2024-12-27T12:41:58Z DEB-NUC11PAH 32727.3038333
2024-12-27T12:42:58Z DEB-NUC11PAH 32787.5
2024-12-27T12:43:58Z DEB-NUC11PAH 32847.5
2024-12-27T12:44:58Z DEB-NUC11PAH 32907.4431667
2024-12-27T12:45:58Z DEB-NUC11PAH 32967.2706667
2024-12-27T12:46:58Z DEB-NUC11PAH 33027.3575
2024-12-27T12:47:58Z DEB-NUC11PAH 33087.5043333
2024-12-27T12:48:58Z DEB-NUC11PAH 33147.4215
2024-12-27T12:49:58Z DEB-NUC11PAH 33207.1713333
```

### **Observations:**
- A data gap is visible in Grafana panels between 12:06 and 12:35, even though the data exists in InfluxDB.
- The time axis appears to have skipped 29 minutes.
- Normal graph rendering resumed after a system reboot.
- The timestamp in the Grafana query is correct.

**Queries are correct, and we also tried to reproduce this issue on a machine with the same configuration, software, and hardware, but there were no time axis gaps in Grafana graphs, even without an internet connection**.

For example purposes, here is the query to verify uptime data:
```
SELECT max("value") FROM "netdata.system.uptime.uptime"
WHERE time >= now() - 6h
GROUP BY time(1m) fill(null)
```

Please check the attached screenshot for reference.
![image](https://github.com/user-attachments/assets/805070af-8bcb-484a-834e-b2543729700f)

### **Environment Details:**
- **OS:** Debian 10 Buster with Backported Kernel 5.10.0-0.deb10.16-amd64
- **Grafana Version:** 7.3.6
- **InfluxDB Version:** 1.8.1
- **Netdata Version:** v1.31.0-68-g2e88104dd

### **Setup:**
- **Data Collection:** Netdata
- **Time-Series Database:** InfluxDB 1.8.1

### **Troubleshooting Steps Taken:**
1. **Verified InfluxDB Data:**
2. **Checked Network & System Uptime:**
- Confirmed that the system was running during the affected period.
3. **Restarted Machine:**
- Around **12:40 PM**, the issue was resolved after **unplugging and reinserting the power cable**, followed by a system reboot.which restored normal functionality.
```
root@DEB-NUC11PAHS:~# last reboot
reboot system boot 5.10.0-0.deb10.1 Tue Jan 21 10:17 still running
reboot system boot 5.10.0-0.deb10.1 Fri Dec 27 12:36 still running

root@DEB-NUC11PAH:~# uptime
20:59:27 up 43 days, 10:42, 5 users, load average: 2.16, 2.03, 2.21
```

### **Additional Information:**
- While the **network was down**, InfluxDB **continued collecting data**, yet Grafana displayed missing data.
- After rebooting the machine, **Grafana started displaying data normally again**.

[1]: https://i.sstatic.net/gvNKrLIz.png

Contributor guide

Open the contributing guide

Research direction

No repository files or tests are named. Start by reproducing the Grafana panel behavior and compare its query results with the provided InfluxDB shell query during the reported time range; done means identifying and fixing the cause of the missing rendered segment, with coverage for the affected case.

Written by the indexing model from the issue text.

Assessment

Tech stack
debian, grafana
Domain
databases, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.