influxdata / influxdata/influxdb
flux: Extremely inefficient when querying the first/last entry
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. create a measurement with a lot of entries
- in total 10000000 entries over 3 year
- 5000000 in this time period are matching the a="b" where clause
2. query last or first entry for measurement - system has extreme high CPU load
__Expected behavior:__
Works as fast as InfluxQL:
# first entry
select x,y,z from foobar where a='b' order by time asc limit 1;
# last entry
select x,y,z from foobar where a='b' order by time desc limit 1;
__Actual behavior:__
A query for the last/first entry require a start range of 0 because it is not previously know. So the whole system spikes in CPU and memory usage and at the end blocks everything.
# first entry
from(bucket:"mybucket")
|> range(start:0)
|> filter(fn: (r) =>
r._measurement == "foobar" and
(r._field == "x" or
r._field == "y" or
r._field == "z") and
r.a == "b")
|> limit(n:1)
|> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
|> group()
|> sort(columns:["_time"], desc:false)
|> limit(n:1)
# last entry
from(bucket:"mybucket")
|> range(start:0)
|> filter(fn: (r) =>
r._measurement == "foobar" and
(r._field == "x" or
r._field == "y" or
r._field == "z") and
r.a == "b")
|> tail(n:1)
|> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
|> group()
|> sort(columns:["_time"], desc:false)
|> tail(n:1)
__Environment info:__
* System info: Linux 4.19.0-13-amd64 x86_64
* InfluxDB version: InfluxDB v1.8.3 (git: 1.8 563e6c3d1a7a2790763c6289501095dbec19244e)
__Config:__
[http]
flux-enabled = true
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Flux queries against a measurement with the described large dataset and compare first/last behavior with the InfluxQL examples. Trace the query path for range(start:0), limit, and tail; done means first/last queries complete without the reported extreme CPU and memory usage.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100