influxdata / influxdata/influxdb

Performance difference between Flux and InfluxQL

Open
#18,088 13 comments 11 reactions 2 assignees View on GitHub

@serenibyss is already working on this.

Since Jul 23, 2021.

area/flux area/performance
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

Hello,

I've recently been doing some tests with influx 1.8 and noticed that Flux queries are considerably slower than their InfluxQL counterparts.

My dataset simulates one month of data every second.
The tests consist of 3 main cases:

  1. Getting the last 7 days of raw data
  2. Getting the last 7 days average with an interval of 5 minutes
  3. Getting the last 30 days average with an interval of 5 minutes

InfluxQL

SELECT value FROM "db"."autogen"."raw.int64" WHERE time > now() - 7d AND time < now() AND "key"='test-key1' AND "acp_name"='test-acp' AND "source"='test-source' FILL(null)

Getting the last 7 days of raw data takes on average 1.8s

SELECT mean("value") as "mean_value" FROM "db"."autogen"."raw.int64" WHERE time > now() - 7d AND time < now() AND "key"='test-key1' AND "acp_name"='test-acp' AND "source"='test-source' GROUP BY time(5m) FILL(null)

Getting the average of the last 7 days with an interval of 5 minutes takes on average 46.3ms

SELECT mean("value") as "mean_value" FROM "db"."autogen"."raw.int64" WHERE time > now() - 30d AND time < now() AND "key"='test-key1' AND "acp_name"='test-acp' AND "source"='test-source' GROUP BY time(5m) FILL(null)

Getting the average of the last 30 days with an interval of 5 minutes takes on average 201.3ms

Flux

from(bucket: "bucket") 
    |> range(start: -7d) 
    |> filter(fn: (r) => r._measurement == "raw.int64" and r._field == "value")
    |> filter(fn: (r) => r.acp_name == "test-acp" and r.key == "test-key1" and r.source == "source") 
    |> keep(columns: ["_time", "_value"])

Getting the last 7 days of raw data takes on average 26.7ms

from(bucket: "bucket") 
    |> range(start: -7d) 
    |> filter(fn: (r) => r._measurement == "raw.int64" and r._field == "value")
    |> filter(fn: (r) => r.acp_name == "test-acp" and r.key == "test-key1" and r.source == "source") 
    |> keep(columns: ["_time", "_value"])
    |> aggregateWindow(every: 5m, fn: mean)

Getting the average of the last 7 days with an interval of 5 minutes takes on average 1.3s

from(bucket: "bucket") 
    |> range(start: -30d) 
    |> filter(fn: (r) => r._measurement == "raw.int64" and r._field == "value")
    |> filter(fn: (r) => r.acp_name == "test-acp" and r.key == "test-key1" and r.source == "source") 
    |> keep(columns: ["_time", "_value"])
    |> aggregateWindow(every: 5m, fn: mean)

Getting the average of the last 30 days with an interval of 5 minutes takes on average 6.2s

The only case where Flux is faster is when getting raw data. If I remove the keep statement the 30d query execution time jumps up to close to 12s.
Also when running the Influx v2 beta these queries are marginally slower.
I understand the interval is small, but still, the performance difference is huge.

I'd like to use some features present in the Influx v2 Go client however (e.g. CSV responses), however the performance penalty is too big to ignore.

Best Regards

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.