influxdata / influxdata/influxdb

Continuous query lost some data

Open
#19,199 0 comments 0 reactions 0 assignees View on GitHub
1.x
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. create one CQ group by 5 seconds for nginx log origin measurement:
CREATE CONTINUOUS QUERY cq_reversepry_request_weekly_5s ON reverseProxyLog
BEGIN
SELECT sum(count) as count, percentile(endDelay, 99) as delay99
INTO reverseProxyLog.weeklyKeep.reversepry_request_weekly_5s
FROM reverseProxyLog.dailyKeep.reversepry_request
GROUP BY time(5s), domain, resultCode, url
END

2. find the count in reversepry_request_weekly_5s is less than reversepry_request
3. check origin data:
> select * from reversepry_request where url = '/test/img/v1/img_0302.png';
name: reversepry_request
time clientIP count domain endDelay host resultCode scheme url
---- -------- ----- ------ -------- ---- ---------- ------ ---
2020-08-01T01:21:31.086Z 10.0.0.0 1 abcdefg.com 0.003 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-01T02:09:38.783Z 1 abcdefg.com 0.002 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-01T03:30:25.288Z 10.0.0.0 1 abcdefg.com 0.004 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-01T03:37:46.61Z 10.0.0.0 1 abcdefg.com 0.002 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-01T05:41:15.283Z 10.0.0.0 1 abcdefg.com 0.002 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-01T08:52:31.939Z 10.0.0.0 1 abcdefg.com 0.003 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-01T09:56:43.516Z 10.0.0.0 1 abcdefg.com 0.008 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-01T12:35:36.663Z 1 abcdefg.com 0.002 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-01T16:09:15.148Z 1 abcdefg.com 0.003 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-02T02:12:47.668Z 10.0.0.0 1 abcdefg.com 0.002 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-02T07:18:57.006Z 10.0.0.0 1 abcdefg.com 0.003 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-02T09:50:44.922Z 10.0.0.0 1 abcdefg.com 0.002 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-02T18:08:19.278Z 10.0.0.0 1 abcdefg.com 0.004 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-03T00:07:39.393Z 10.0.0.0 1 abcdefg.com 0.003 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-03T01:00:34.176Z 1 abcdefg.com 0.003 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-03T02:05:49.285Z 10.0.0.0 1 abcdefg.com 0.003 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-03T03:10:58.512Z 10.0.0.0 1 abcdefg.com 0.003 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-03T03:16:05.712Z 10.0.0.0 1 abcdefg.com 0.003 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-03T03:25:08.112Z 10.0.0.0 1 abcdefg.com 0.002 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-03T04:53:36.52Z 10.0.0.0 1 abcdefg.com 0.003 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-03T06:37:04.957Z 10.0.0.0 1 abcdefg.com 0.003 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-03T06:38:49.316Z 10.0.0.0 1 abcdefg.com 0.003 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-03T08:17:49.236Z 1 abcdefg.com 0.004 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-03T09:20:20.123Z 1 abcdefg.com 0.004 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-03T12:15:50.158Z 10.0.0.0 1 abcdefg.com 0.003 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-03T23:37:46.718Z 1 abcdefg.com 0.003 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-04T01:18:00.335Z 10.0.0.0 1 abcdefg.com 0.002 lfgpr05-in 200 https /test/img/v1/img_0302.png
2020-08-04T01:21:30.455Z 10.0.0.0 1 abcdefg.com 0 lfgpr05-in 301 http /test/img/v1/img_0302.png
**2020-08-04T01:21:30.579Z 10.0.0.0 1 abcdefg.com 0.007 lfgpr05-in 200 https /test/img/v1/img_0302.png**

> select * from weeklyKeep.reversepry_request_weekly_5s where time > now() - 3m and url = '/test/img/v1/img_0302.png';
name: reversepry_request_weekly_5s
time count delay99 domain resultCode url
---- ----- ------- ------ ---------- ---
2020-08-04T01:21:30Z 1 0 abcdefg.com 301 /xinghui/img/v1/img_0302.png
**2020-08-04T01:21:30Z 0.007 abcdefg.com 200 /xinghui/img/v1/img_0302.png**
>

__Expected behavior:__
I expected count = 1 in reversepry_request_weekly_5s when resultCode = 200

__Actual behavior:__
But it is blank

__Environment info:__

* System info: Linux 3.10.0-862.14.4.el7.x86_64 x86_64
* InfluxDB version: InfluxDB v1.8.0 (git: 1.8 781490de48220d7695a05c29e5a36f550a4568f5)
* Other relevant environment details: not continaer, binary file runs directly

__Config:__
dir = "/home/influxdb/data"
wal-dir = "/home/influxdb/wal"
max-series-per-database = 5000000
max-values-per-tag = 500000

__Logs:__
Include snippet of errors in log.

__Performance:__
CPU, Mem, and IO all are OK, the load is very low

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the continuous query with the provided source and destination queries, then compare the source rows with the grouped results for the 200 response. No source file or test is named; the issue is done when the continuous query preserves the expected count for each matching row and a regression check covers the case.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.