influxdata / influxdata/influxdb
Performance difference between Flux and InfluxQL
@serenibyss is already working on this.
Since Jul 23, 2021.
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Hello,
I've recently been doing some tests with influx 1.8 and noticed that Flux queries are considerably slower than their InfluxQL counterparts.
My dataset simulates one month of data every second.
The tests consist of 3 main cases:
- Getting the last 7 days of raw data
- Getting the last 7 days average with an interval of 5 minutes
- Getting the last 30 days average with an interval of 5 minutes
InfluxQL
SELECT value FROM "db"."autogen"."raw.int64" WHERE time > now() - 7d AND time < now() AND "key"='test-key1' AND "acp_name"='test-acp' AND "source"='test-source' FILL(null)
Getting the last 7 days of raw data takes on average 1.8s
SELECT mean("value") as "mean_value" FROM "db"."autogen"."raw.int64" WHERE time > now() - 7d AND time < now() AND "key"='test-key1' AND "acp_name"='test-acp' AND "source"='test-source' GROUP BY time(5m) FILL(null)
Getting the average of the last 7 days with an interval of 5 minutes takes on average 46.3ms
SELECT mean("value") as "mean_value" FROM "db"."autogen"."raw.int64" WHERE time > now() - 30d AND time < now() AND "key"='test-key1' AND "acp_name"='test-acp' AND "source"='test-source' GROUP BY time(5m) FILL(null)
Getting the average of the last 30 days with an interval of 5 minutes takes on average 201.3ms
Flux
from(bucket: "bucket")
|> range(start: -7d)
|> filter(fn: (r) => r._measurement == "raw.int64" and r._field == "value")
|> filter(fn: (r) => r.acp_name == "test-acp" and r.key == "test-key1" and r.source == "source")
|> keep(columns: ["_time", "_value"])
Getting the last 7 days of raw data takes on average 26.7ms
from(bucket: "bucket")
|> range(start: -7d)
|> filter(fn: (r) => r._measurement == "raw.int64" and r._field == "value")
|> filter(fn: (r) => r.acp_name == "test-acp" and r.key == "test-key1" and r.source == "source")
|> keep(columns: ["_time", "_value"])
|> aggregateWindow(every: 5m, fn: mean)
Getting the average of the last 7 days with an interval of 5 minutes takes on average 1.3s
from(bucket: "bucket")
|> range(start: -30d)
|> filter(fn: (r) => r._measurement == "raw.int64" and r._field == "value")
|> filter(fn: (r) => r.acp_name == "test-acp" and r.key == "test-key1" and r.source == "source")
|> keep(columns: ["_time", "_value"])
|> aggregateWindow(every: 5m, fn: mean)
Getting the average of the last 30 days with an interval of 5 minutes takes on average 6.2s
The only case where Flux is faster is when getting raw data. If I remove the keep statement the 30d query execution time jumps up to close to 12s.
Also when running the Influx v2 beta these queries are marginally slower.
I understand the interval is small, but still, the performance difference is huge.
I'd like to use some features present in the Influx v2 Go client however (e.g. CSV responses), however the performance penalty is too big to ignore.
Best Regards
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.