influxdata / influxdata/influxdb

Optimize a flux script that uses a "static" function (the query planner could replace the function by some static equivalent instructions)

Open
#19,257 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

team/query
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

@AnesBendimerad commented on [Fri Aug 07 2020](https://github.com/influxdata/flux/issues/3084)

When a function is used in a flux script, it will add a significant cost on the performance of the query. For example, let's compare the two scripts that do the same thing, but the second one uses the function "contains":
1. Script without "contains":
```
from(bucket: "Metrics")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "MetricsMachine")
|> filter(fn: (r) => r["_field"] == "memEchange" or r["_field"] == "chargeCpu")
|> aggregateWindow(every: 30m, fn: mean)
```

2. Script with "contains":
```
from(bucket: "Metrics")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "MetricsMachine")
|> filter(fn: (r) => contains(value: r["_field"], set: ["memEchange","chargeCpu"]))
|> aggregateWindow(every: 30m, fn: mean)
```
The first script takes 10 seconds, while the second script takes 20 seconds (10 seconds more, because of using "contains" when filtering the "_field" column).

Using a function that performs a static operation (as I did with "contains") can be helpful in many situations, to structure the flux script and make it more readable. However it adds a significant cost on the computation time.

So, it would be very helpful to add the following feature:
Make the query planner able to recognize functions with "static" operations and optimize them. For example, the planner would be able to replace my usage of ` contains(value: r["_field"], set: ["memEchange","chargeCpu"])` with an "or" condition: ` r["_field"] == "memEchange" or r["_field"] == "chargeCpu"`.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the two Flux scripts in the issue and comparing their query performance, using the reported contains-based filter as the reproduction. Read the query-planning path that handles function calls; done means the planner recognizes this static operation and produces equivalent optimized behavior without the reported performance cost.

Written by the indexing model from the issue text.

Assessment

Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.