influxdata / influxdata/influxdb

Data out of order after 1.8.9 to 2.0.8 upgrade with flux queries

Open
#22,436 1 comment 0 reactions 0 assignees View on GitHub
area/2.x area/compat-v1x area/flux kind/bug
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. Existing DB in influx 1.8.9
2. Automatic upgrade to 2.0.8
3. Run flux query

```
from(bucket: "bucket/autogen")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "rht" and r["nid"] == "35")
|> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")
|> drop(columns: ["_start", "_stop", "_measurement"])
|> rename(columns: {_time: "time"})
```

__Expected behavior:__

Ordered data in time as previously with the Flux query on 1.8.9

__Actual behavior:__

Out of order data - some data are added at the end of the output

We had to add a sort() call to get same output as before.

```
from(bucket: "bucket/autogen")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "rht" and r["nid"] == "35")
|> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")
|> drop(columns: ["_start", "_stop", "_measurement"])
|> rename(columns: {_time: "time"})
|> sort(columns: ["time"], desc: false)
```

If we sort() (ie `|> sort(columns: ["_time"], desc: false)`before `pivot()`, data are still out of order at the end of the query.

As you can see in attached json files (generated with PHP Client 2.1.0 - but we have the same output in InfluxDB2 UI):
* v1 output : all entries are ordred by date
* v2 output : at the end, you will see old data

[flux_query_output_v1.txt](https://github.com/influxdata/influxdb/files/7143076/flux_query_output_v1.txt)
[flux_query_output_v2.txt](https://github.com/influxdata/influxdb/files/7143077/flux_query_output_v2.txt)

__Environment info:__

* System info: CentOS 7.9
* InfluxDB version: InfluxDB 2.0.8

Questions on our side:
* After upgrade, can we enforce a bucket reindexation so that data are ordered again ? Or do we have to plan to do a post upgrade tasks which will rewrite all data in new buckets ?
* We would prefer to avoid adding the `sort()` call everywhere in our apps. Otherwise what's the overhead of adding the `sort()` call to all our queries ?

Contributor guide

Open the contributing guide

Research direction

No source file or test is named. Start by reproducing the 1.8.9-to-2.0.8 upgrade and running the provided Flux query, then compare the attached v1 and v2 outputs; done means determining whether upgraded data can preserve ordering without adding sort() to the query.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.