influxdata / influxdata/influxdb

applying aggregateWindow twice results in out-of-order and duplicate results

Open
#23,962 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

You should be able to use to following, kind of minimal example. Note that this example by itself does not make any sense, as it is reduced to a very simple example. At the same time, this short example is good to show that something is wrong here.

```
import "influxdata/influxdb/sample"
import "date"

Monday = date.add(d: 4d, to: date.truncate(t: date.sub(d: 4d, from: now()), unit: 1w))

sample.data(set: "airSensor")
|> range(start: date.sub(d:9d, from: Monday),
stop: Monday)
|> aggregateWindow(every: 1h, fn: last)
|> aggregateWindow(every: 1d, fn: last)
```

__Expected behavior:__
I expect the first aggregateWindow to bin results into 1 hour intervals, using the last value in that bin, and the second aggregateWindow to do the same but daily, and using the bin values from the first aggregateWindow as input.

__Actual behavior:__
This is what I get as the first rows:

```
0 airSensors co 2022-11-27T00:00:00.000Z TLM0100
0 airSensors co 2022-12-05T00:00:00.000Z TLM0100
0 airSensors co 2022-11-28T00:00:00.000Z TLM0100
0 airSensors co 2022-11-29T00:00:00.000Z TLM0100
0 airSensors co 2022-11-30T00:00:00.000Z TLM0100
0 airSensors co 2022-12-01T00:00:00.000Z TLM0100
0 airSensors co 2022-12-02T00:00:00.000Z TLM0100
0 airSensors co 2022-12-03T00:00:00.000Z TLM0100
0 airSensors co 2022-12-04T00:00:00.000Z TLM0100
0 airSensors co 2022-12-05T00:00:00.000Z TLM0100
```

Note the time stamp of the second row. Not only is it out of order with respect to time, it is also duplicated at the end. Also, not visible in this example, on my data, the values can actually differ (slightly) for the two duplicates (timestamp is the same, but the measurement value is not).

I am not sure if this is related to #21111, but since there it is the first value that is wrong and here the second, I rather open a new issue and point to possibly related ones.

__Environment info:__

* System info: Linux 5.15.76-v8+ aarch64
* InfluxDB version: InfluxDB v2.5.1 (git: 5b6fdbf05d) build_date: 2022-11-02T18:06:20Z
* Other relevant environment details: Debian package

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied Flux query against InfluxDB v2.5.1 and inspect the ordering and duplicate timestamps in its output. Compare the behavior with issue #21111, which the report identifies as possibly related. Done means repeated aggregateWindow calls preserve daily result ordering and do not emit duplicate timestamps, with a regression test for the example.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.