influxdata / influxdata/influxdb
Optimize a flux script that uses a "static" function (the query planner could replace the function by some static equivalent instructions)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
@AnesBendimerad commented on [Fri Aug 07 2020](https://github.com/influxdata/flux/issues/3084)
When a function is used in a flux script, it will add a significant cost on the performance of the query. For example, let's compare the two scripts that do the same thing, but the second one uses the function "contains":
1. Script without "contains":
```
from(bucket: "Metrics")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "MetricsMachine")
|> filter(fn: (r) => r["_field"] == "memEchange" or r["_field"] == "chargeCpu")
|> aggregateWindow(every: 30m, fn: mean)
```
2. Script with "contains":
```
from(bucket: "Metrics")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "MetricsMachine")
|> filter(fn: (r) => contains(value: r["_field"], set: ["memEchange","chargeCpu"]))
|> aggregateWindow(every: 30m, fn: mean)
```
The first script takes 10 seconds, while the second script takes 20 seconds (10 seconds more, because of using "contains" when filtering the "_field" column).
Using a function that performs a static operation (as I did with "contains") can be helpful in many situations, to structure the flux script and make it more readable. However it adds a significant cost on the computation time.
So, it would be very helpful to add the following feature:
Make the query planner able to recognize functions with "static" operations and optimize them. For example, the planner would be able to replace my usage of ` contains(value: r["_field"], set: ["memEchange","chargeCpu"])` with an "or" condition: ` r["_field"] == "memEchange" or r["_field"] == "chargeCpu"`.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the two Flux scripts in the issue and comparing their query performance, using the reported contains-based filter as the reproduction. Read the query-planning path that handles function calls; done means the planner recognizes this static operation and produces equivalent optimized behavior without the reported performance cost.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100