influxdata / influxdata/influxdb

flux: Extremely inefficient when querying the first/last entry

Open
#20,626 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. create a measurement with a lot of entries

- in total 10000000 entries over 3 year
- 5000000 in this time period are matching the a="b" where clause

2. query last or first entry for measurement - system has extreme high CPU load

__Expected behavior:__

Works as fast as InfluxQL:

# first entry
select x,y,z from foobar where a='b' order by time asc limit 1;

# last entry
select x,y,z from foobar where a='b' order by time desc limit 1;

__Actual behavior:__

A query for the last/first entry require a start range of 0 because it is not previously know. So the whole system spikes in CPU and memory usage and at the end blocks everything.

# first entry
from(bucket:"mybucket")
|> range(start:0)
|> filter(fn: (r) =>
r._measurement == "foobar" and
(r._field == "x" or
r._field == "y" or
r._field == "z") and
r.a == "b")
|> limit(n:1)
|> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
|> group()
|> sort(columns:["_time"], desc:false)
|> limit(n:1)

# last entry
from(bucket:"mybucket")
|> range(start:0)
|> filter(fn: (r) =>
r._measurement == "foobar" and
(r._field == "x" or
r._field == "y" or
r._field == "z") and
r.a == "b")
|> tail(n:1)
|> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
|> group()
|> sort(columns:["_time"], desc:false)
|> tail(n:1)

__Environment info:__

* System info: Linux 4.19.0-13-amd64 x86_64
* InfluxDB version: InfluxDB v1.8.3 (git: 1.8 563e6c3d1a7a2790763c6289501095dbec19244e)

__Config:__

[http]
flux-enabled = true

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Flux queries against a measurement with the described large dataset and compare first/last behavior with the InfluxQL examples. Trace the query path for range(start:0), limit, and tail; done means first/last queries complete without the reported extreme CPU and memory usage.

Written by the indexing model from the issue text.

Assessment

Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.