influxdata / influxdata/influxdb
Data out of order after 1.8.9 to 2.0.8 upgrade with flux queries
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. Existing DB in influx 1.8.9
2. Automatic upgrade to 2.0.8
3. Run flux query
```
from(bucket: "bucket/autogen")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "rht" and r["nid"] == "35")
|> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")
|> drop(columns: ["_start", "_stop", "_measurement"])
|> rename(columns: {_time: "time"})
```
__Expected behavior:__
Ordered data in time as previously with the Flux query on 1.8.9
__Actual behavior:__
Out of order data - some data are added at the end of the output
We had to add a sort() call to get same output as before.
```
from(bucket: "bucket/autogen")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "rht" and r["nid"] == "35")
|> pivot(rowKey: ["_time"], columnKey: ["_field"], valueColumn: "_value")
|> drop(columns: ["_start", "_stop", "_measurement"])
|> rename(columns: {_time: "time"})
|> sort(columns: ["time"], desc: false)
```
If we sort() (ie `|> sort(columns: ["_time"], desc: false)`before `pivot()`, data are still out of order at the end of the query.
As you can see in attached json files (generated with PHP Client 2.1.0 - but we have the same output in InfluxDB2 UI):
* v1 output : all entries are ordred by date
* v2 output : at the end, you will see old data
[flux_query_output_v1.txt](https://github.com/influxdata/influxdb/files/7143076/flux_query_output_v1.txt)
[flux_query_output_v2.txt](https://github.com/influxdata/influxdb/files/7143077/flux_query_output_v2.txt)
__Environment info:__
* System info: CentOS 7.9
* InfluxDB version: InfluxDB 2.0.8
Questions on our side:
* After upgrade, can we enforce a bucket reindexation so that data are ordered again ? Or do we have to plan to do a post upgrade tasks which will rewrite all data in new buckets ?
* We would prefer to avoid adding the `sort()` call everywhere in our apps. Otherwise what's the overhead of adding the `sort()` call to all our queries ?
Contributor guide
Research direction
No source file or test is named. Start by reproducing the 1.8.9-to-2.0.8 upgrade and running the provided Flux query, then compare the attached v1 and v2 outputs; done means determining whether upgraded data can preserve ordering without adding sort() to the query.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100