influxdata / influxdata/influxdb
non_negative_derivative and group by sizing causing large spikes
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Hi,
We're using InfluxDB 0.12.0 and Grafana 3.0.0 beta31460390657. We're monitoring systems that output continually increasing counters (Lustre) via Telegraf. The metrics are being collected by Telegraf every 10s.
For smaller time ranges the graphs are accurate. However for larger time periods InfluxDB sends back data with a very large spike at the beginning. I've noticed this seems to stem from the larger `GROUP BY time()` values that Grafana selects when you view data over a larger time period.
If I manually select a `GROUP BY time(10s)` the data is always correct. If I let Grafana auto-select with a `GROUP BY time($interval)`, I see the issue. Note that the issue is periodic - sometimes the large interval works and sometimes it doesn't. I'm guessing this is because I'm using `now()` in my queries so at certain times the times align correctly.
So to cut Grafana out of the equation I've manually run the queries to illustrate the issue. If I use `GROUP BY time(10s)`, this query always returns correct data no matter how many times I try it:
```
> SELECT non_negative_derivative(sum("read_bytes"), 1s) FROM "lustre2" WHERE time > now() - 6h GROUP BY time(10s) fill(null) LIMIT 10;
name: lustre2
-------------
time non_negative_derivative
1461153160000000000 0
1461153170000000000 0
1461153180000000000 0
1461153190000000000 0
1461153200000000000 0
1461153210000000000 0
1461153220000000000 0
1461153230000000000 0
1461153240000000000 0
1461153250000000000 0
```
However, if I increase to `GROUP BY time(30s)` and run two queries that were less than a second apart:
```
> SELECT non_negative_derivative(sum("read_bytes"), 1s) FROM "lustre2" WHERE time > now() - 6h GROUP BY time(30s) fill(null) LIMIT 10;
name: lustre2
-------------
time non_negative_derivative
1461153210000000000 0
1461153240000000000 0
1461153270000000000 0
1461153300000000000 0
1461153330000000000 0
1461153360000000000 0
1461153390000000000 0
1461153420000000000 0
1461153450000000000 0
1461153480000000000 0
> SELECT non_negative_derivative(sum("read_bytes"), 1s) FROM "lustre2" WHERE time > now() - 6h GROUP BY time(30s) fill(null) LIMIT 10;
name: lustre2
-------------
time non_negative_derivative
1461153210000000000 4.387031397146e+11
1461153240000000000 0
1461153270000000000 0
1461153300000000000 0
1461153330000000000 0
1461153360000000000 0
1461153390000000000 0
1461153420000000000 0
1461153450000000000 0
1461153480000000000 0
```
In fact if I run this query every second, in seconds 1 - 10 the first value is 0, in seconds 11-20, the first value is a very large number, and in seconds 21 - 30, the first value is an even larger number (almost exactly double). Then the first value returns to 0 after that and the cycle starts again. This is due to the sum values being very different (three queries run with about 10s in between):
```
> SELECT sum("read_bytes") FROM "lustre2" WHERE time > now() - 6h GROUP BY time(30s) LIMIT 10;
name: lustre2
-------------
time sum
1461153630000000000 13161094191438
1461153660000000000 39483282574314
1461153690000000000 39483282574314
1461153720000000000 39483282574314
1461153750000000000 39483282574314
1461153780000000000 39483282574314
1461153810000000000 39483282574314
1461153840000000000 39483282574314
1461153870000000000 39483282574314
1461153900000000000 39483282574314
> SELECT sum("read_bytes") FROM "lustre2" WHERE time > now() - 6h GROUP BY time(30s) LIMIT 10;
name: lustre2
-------------
time sum
1461153660000000000 26322188382876
1461153690000000000 39483282574314
1461153720000000000 39483282574314
1461153750000000000 39483282574314
1461153780000000000 39483282574314
1461153810000000000 39483282574314
1461153840000000000 39483282574314
1461153870000000000 39483282574314
1461153900000000000 39483282574314
1461153930000000000 39483282574314
> SELECT sum("read_bytes") FROM "lustre2" WHERE time > now() - 6h GROUP BY time(30s) LIMIT 10;
name: lustre2
-------------
time sum
1461153660000000000
1461153690000000000 39483282574314
1461153720000000000 39483282574314
1461153750000000000 39483282574314
1461153780000000000 39483282574314
1461153810000000000 39483282574314
1461153840000000000 39483282574314
1461153870000000000 39483282574314
1461153900000000000 39483282574314
1461153930000000000 39483282574314
```
When I look at the raw data points they are staying constant as expected since there is no activity right now (using one host as an example):
```
> SELECT read_bytes FROM "lustre2" WHERE "host"='hpcleo00' GROUP BY "name" LIMIT 10;
name: lustre2
tags: name=lustree-OST0000
time read_bytes
---- ----------
1459977920000000000 39325212672
1459977930000000000 39325212672
1459977940000000000 39325212672
1459977950000000000 39325212672
1459977960000000000 39325212672
1459977970000000000 39325212672
1459977980000000000 39325212672
1459977990000000000 39325212672
1459978000000000000 39325212672
1459978010000000000 39325212672
name: lustre2
tags: name=lustree-OST0001
time read_bytes
---- ----------
1459977920000000000 39718395904
1459977930000000000 39718395904
1459977940000000000 39718395904
1459977950000000000 39718395904
1459977960000000000 39718395904
1459977970000000000 39718395904
1459977980000000000 39718395904
1459977990000000000 39718395904
1459978000000000000 39718395904
1459978010000000000 39718395904
name: lustre2
tags: name=lustree-OST0002
time read_bytes
---- ----------
1459977920000000000 59411320832
1459977930000000000 59411320832
1459977940000000000 59411320832
1459977950000000000 59411320832
1459977960000000000 59411320832
1459977970000000000 59411320832
1459977980000000000 59411320832
1459977990000000000 59411320832
1459978000000000000 59411320832
1459978010000000000 59411320832
name: lustre2
tags: name=lustree-OST0003
time read_bytes
---- ----------
1459977920000000000 59857276928
1459977930000000000 59857276928
1459977940000000000 59857276928
1459977950000000000 59857276928
1459977960000000000 59857276928
1459977970000000000 59857276928
1459977980000000000 59857276928
1459977990000000000 59857276928
1459978000000000000 59857276928
1459978010000000000 59857276928
name: lustre2
tags: name=lustree-OST0004
time read_bytes
---- ----------
1459977920000000000 39734607872
1459977930000000000 39734607872
1459977940000000000 39734607872
1459977950000000000 39734607872
1459977960000000000 39734607872
1459977970000000000 39734607872
1459977980000000000 39734607872
1459977990000000000 39734607872
1459978000000000000 39734607872
1459978010000000000 39734607872
```
Note that I have also tried all the fill options (none, null, 0, previous) and they do not help.
I've read through https://github.com/influxdata/influxdb/issues/4237, https://github.com/influxdata/influxdb/pull/4292, and all the linked issues, but I think this is different.
Thanks,
Shawn
Contributor guide
Research direction
Start by reproducing the two GROUP BY time(10s) and time(30s) queries against the reported counter data, then trace the query path for sum() and non_negative_derivative() with time bucketing. Compare results as now() moves through the 30-second cycle. Done means repeated equivalent queries no longer produce a changing or oversized first value, with regression coverage for the reported case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100