influxdata / influxdata/influxdb

non_negative_derivative and group by sizing causing large spikes

Open
#6,451 38 comments 19 reactions 0 assignees View on GitHub
1.x area/queries flux/triaged
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Hi,
We're using InfluxDB 0.12.0 and Grafana 3.0.0 beta31460390657. We're monitoring systems that output continually increasing counters (Lustre) via Telegraf. The metrics are being collected by Telegraf every 10s.

For smaller time ranges the graphs are accurate. However for larger time periods InfluxDB sends back data with a very large spike at the beginning. I've noticed this seems to stem from the larger `GROUP BY time()` values that Grafana selects when you view data over a larger time period.

If I manually select a `GROUP BY time(10s)` the data is always correct. If I let Grafana auto-select with a `GROUP BY time($interval)`, I see the issue. Note that the issue is periodic - sometimes the large interval works and sometimes it doesn't. I'm guessing this is because I'm using `now()` in my queries so at certain times the times align correctly.

So to cut Grafana out of the equation I've manually run the queries to illustrate the issue. If I use `GROUP BY time(10s)`, this query always returns correct data no matter how many times I try it:

```
> SELECT non_negative_derivative(sum("read_bytes"), 1s) FROM "lustre2" WHERE time > now() - 6h GROUP BY time(10s) fill(null) LIMIT 10;
name: lustre2
-------------
time non_negative_derivative
1461153160000000000 0
1461153170000000000 0
1461153180000000000 0
1461153190000000000 0
1461153200000000000 0
1461153210000000000 0
1461153220000000000 0
1461153230000000000 0
1461153240000000000 0
1461153250000000000 0
```

However, if I increase to `GROUP BY time(30s)` and run two queries that were less than a second apart:

```
> SELECT non_negative_derivative(sum("read_bytes"), 1s) FROM "lustre2" WHERE time > now() - 6h GROUP BY time(30s) fill(null) LIMIT 10;
name: lustre2
-------------
time non_negative_derivative
1461153210000000000 0
1461153240000000000 0
1461153270000000000 0
1461153300000000000 0
1461153330000000000 0
1461153360000000000 0
1461153390000000000 0
1461153420000000000 0
1461153450000000000 0
1461153480000000000 0

> SELECT non_negative_derivative(sum("read_bytes"), 1s) FROM "lustre2" WHERE time > now() - 6h GROUP BY time(30s) fill(null) LIMIT 10;
name: lustre2
-------------
time non_negative_derivative
1461153210000000000 4.387031397146e+11
1461153240000000000 0
1461153270000000000 0
1461153300000000000 0
1461153330000000000 0
1461153360000000000 0
1461153390000000000 0
1461153420000000000 0
1461153450000000000 0
1461153480000000000 0
```

In fact if I run this query every second, in seconds 1 - 10 the first value is 0, in seconds 11-20, the first value is a very large number, and in seconds 21 - 30, the first value is an even larger number (almost exactly double). Then the first value returns to 0 after that and the cycle starts again. This is due to the sum values being very different (three queries run with about 10s in between):

```
> SELECT sum("read_bytes") FROM "lustre2" WHERE time > now() - 6h GROUP BY time(30s) LIMIT 10;
name: lustre2
-------------
time sum
1461153630000000000 13161094191438
1461153660000000000 39483282574314
1461153690000000000 39483282574314
1461153720000000000 39483282574314
1461153750000000000 39483282574314
1461153780000000000 39483282574314
1461153810000000000 39483282574314
1461153840000000000 39483282574314
1461153870000000000 39483282574314
1461153900000000000 39483282574314

> SELECT sum("read_bytes") FROM "lustre2" WHERE time > now() - 6h GROUP BY time(30s) LIMIT 10;
name: lustre2
-------------
time sum
1461153660000000000 26322188382876
1461153690000000000 39483282574314
1461153720000000000 39483282574314
1461153750000000000 39483282574314
1461153780000000000 39483282574314
1461153810000000000 39483282574314
1461153840000000000 39483282574314
1461153870000000000 39483282574314
1461153900000000000 39483282574314
1461153930000000000 39483282574314

> SELECT sum("read_bytes") FROM "lustre2" WHERE time > now() - 6h GROUP BY time(30s) LIMIT 10;
name: lustre2
-------------
time sum
1461153660000000000
1461153690000000000 39483282574314
1461153720000000000 39483282574314
1461153750000000000 39483282574314
1461153780000000000 39483282574314
1461153810000000000 39483282574314
1461153840000000000 39483282574314
1461153870000000000 39483282574314
1461153900000000000 39483282574314
1461153930000000000 39483282574314
```

When I look at the raw data points they are staying constant as expected since there is no activity right now (using one host as an example):

```
> SELECT read_bytes FROM "lustre2" WHERE "host"='hpcleo00' GROUP BY "name" LIMIT 10;
name: lustre2
tags: name=lustree-OST0000
time read_bytes
---- ----------
1459977920000000000 39325212672
1459977930000000000 39325212672
1459977940000000000 39325212672
1459977950000000000 39325212672
1459977960000000000 39325212672
1459977970000000000 39325212672
1459977980000000000 39325212672
1459977990000000000 39325212672
1459978000000000000 39325212672
1459978010000000000 39325212672

name: lustre2
tags: name=lustree-OST0001
time read_bytes
---- ----------
1459977920000000000 39718395904
1459977930000000000 39718395904
1459977940000000000 39718395904
1459977950000000000 39718395904
1459977960000000000 39718395904
1459977970000000000 39718395904
1459977980000000000 39718395904
1459977990000000000 39718395904
1459978000000000000 39718395904
1459978010000000000 39718395904

name: lustre2
tags: name=lustree-OST0002
time read_bytes
---- ----------
1459977920000000000 59411320832
1459977930000000000 59411320832
1459977940000000000 59411320832
1459977950000000000 59411320832
1459977960000000000 59411320832
1459977970000000000 59411320832
1459977980000000000 59411320832
1459977990000000000 59411320832
1459978000000000000 59411320832
1459978010000000000 59411320832
name: lustre2
tags: name=lustree-OST0003
time read_bytes
---- ----------
1459977920000000000 59857276928
1459977930000000000 59857276928
1459977940000000000 59857276928
1459977950000000000 59857276928
1459977960000000000 59857276928
1459977970000000000 59857276928
1459977980000000000 59857276928
1459977990000000000 59857276928
1459978000000000000 59857276928
1459978010000000000 59857276928

name: lustre2
tags: name=lustree-OST0004
time read_bytes
---- ----------
1459977920000000000 39734607872
1459977930000000000 39734607872
1459977940000000000 39734607872
1459977950000000000 39734607872
1459977960000000000 39734607872
1459977970000000000 39734607872
1459977980000000000 39734607872
1459977990000000000 39734607872
1459978000000000000 39734607872
1459978010000000000 39734607872
```

Note that I have also tried all the fill options (none, null, 0, previous) and they do not help.

I've read through https://github.com/influxdata/influxdb/issues/4237, https://github.com/influxdata/influxdb/pull/4292, and all the linked issues, but I think this is different.

Thanks,
Shawn

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the two GROUP BY time(10s) and time(30s) queries against the reported counter data, then trace the query path for sum() and non_negative_derivative() with time bucketing. Compare results as now() moves through the 30-second cycle. Done means repeated equivalent queries no longer produce a changing or oversized first value, with regression coverage for the reported case.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.