influxdata / influxdata/influxdb

Inconsistent/missing Query Data

Open
#18,870 3 comments 1 reaction 1 assignee Claimed by @dgnorton View on GitHub
1.x
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

**System Info:** 64 bit Ubuntu 14.04.4
**InfluxDB version:** OSS 1.2.4 installed via dpkg
**Retention Policy:** default -- (autogen | "0s" | "168h0m0s" | 1 | true)

**Problem Summary:**
Running the same query multiple times will sometimes return data and sometimes it will not. Over time the returned data will flip-flop, meaning the data that did not return running the query previously now does return and vice-versa.
For example:
Query 1: `select value from ethRxFrameDataRate where model='MX67' and test_run = '4019485'`
Query 2: `select value from ethRxFrameDataRate where model='MX100' and test_run = '4019485'`
Initially Query 1 returns data:
```
influxdb.query("select value from ethRxFrameDataRate where model='MX67' and test_run = '4019485'")
=> [{"name"=>"ethRxFrameDataRate", "tags"=>nil, "values"=>[{"time"=>"2020-06-26T14:40:37Z", "value"=> ...
...
```
and Query 2 does not:
```
influxdb.query("select value from ethRxFrameDataRate where model='MX100' and test_run = '4019485'")
=> []
```
A few minutes later we see...
Query 1 no longer returns data:
```
influxdb.query("select value from ethRxFrameDataRate where model='MX67' and test_run = '4019485'")
=> []
```
but Query 2 does return data:
```
influxdb.query("select value from ethRxFrameDataRate where model='MX100' and test_run = '4019485'")
=> [{"name"=>"ethRxFrameDataRate", "tags"=>nil, "values"=>[{"time"=>"2020-06-26T12:26:45Z", "value"=> ...
...
```
Another interesting observation is that running the same queries from influx console I have only ever observed Query 2 return data (even when the external query is returning data for Query 1).
Also, I have **never** been able to get data for both queries at the same time.

**Troubleshooting Steps:**

- I backed up the database using `influxd backup` and restored to a new instance of influx 1.2.4 on another Ubuntu 14.04 server using `influxd restore` and the same issue is seen
- I restored that same backup to a server running influx 1.8.0 and the same behavior is seen
- Looking at the logs it seems that it _may_ be related to when compaction occurs. I've seen the data flip-flop after seeing the following in the logs:
```
[I] 2020-07-06T21:44:54Z Snapshot for path /var/lib/influxdb/data/_internal/monitor/2351 written in 336.902476ms engine=tsm1
[I] 2020-07-06T21:53:54Z Snapshot for path /var/lib/influxdb/data/_internal/monitor/2351 written in 299.038848ms engine=tsm1
[I] 2020-07-06T21:53:55Z beginning level 1 compaction of group 0, 2 TSM files engine=tsm1
[I] 2020-07-06T21:53:55Z compacting level 1 group (0) /var/lib/influxdb/data/_internal/monitor/2351/000000145-000000001.tsm (#0) engine=tsm1
[I] 2020-07-06T21:53:55Z compacting level 1 group (0) /var/lib/influxdb/data/_internal/monitor/2351/000000146-000000001.tsm (#1) engine=tsm1
[I] 2020-07-06T21:53:56Z compacted level 1 group (0) into /var/lib/influxdb/data/_internal/monitor/2351/000000146-000000002.tsm.tmp (#0) engine=tsm1
[I] 2020-07-06T21:53:56Z compacted level 1 2 files into 1 files in 834.951726ms engine=tsm1
```
However, this is somewhat anecdotal and I don't have evidence that this is occurring at every compaction.

**Questions:**
Are there any known issues that could cause behavior like this (I searched through issues and did not find any)? What other logs can I pull or what debug can be enabled to provide more insight into this issue?

This is impacting some critical production data so any help would be appreciated!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.