influxdata / influxdata/influxdb

The contains function may not be optimized

Open
#20,931 2 comments 3 reactions 0 assignees View on GitHub
area/2.x area/flux area/performance
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

### Environment info:
**influxDB version**: 2.0.3

**System info**: from docker
Debain, X86_64, 8-core Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz, 16GB RAM

### Data describe:
**BucketName**: 15day_profile_bucket
**MeasurementName**: function_info
**Tags**: Function, Pid, Tid, ProcessName, UUID, State
**Fields**: Internal, cumulative

there may be 2.4w record and 200 series in 1 minute.
### Problem:
The `contains` function query is very slow, it seems that the group key filter is not used.

The following flux query took 0.63s:

```
from(bucket: "15day_profile_bucket")
|> range(start: 2021-03-03T05:54:39.611Z, stop: 2021-03-03T06:54:39.611Z)
|> filter(fn: (r) => r["_measurement"] == "function_info" )
|> limit(n: 1)
|> filter(fn: (r) => contains(value: r["UUID"], set: ["7f0a1436-37ad-4b7a-9ab1-7acce9ee3060"]) )
|> yield()
```

this is image:
![image](https://user-images.githubusercontent.com/42167339/111055963-1a97c380-84b6-11eb-8d7a-999222401340.png)

but the flux query took 37.88s:
```
from(bucket: "15day_profile_bucket")
|> range(start: 2021-03-03T05:54:39.611Z, stop: 2021-03-03T06:54:39.611Z)
|> filter(fn: (r) => r["_measurement"] == "function_info" )
|> filter(fn: (r) => contains(value: r["UUID"], set: ["7f0a1436-37ad-4b7a-9ab1-7acce9ee3060"]) )
|> limit(n: 1)
|> yield()
```

this is image:

![image](https://user-images.githubusercontent.com/42167339/110574280-cb8d1e00-8197-11eb-86df-96fd8aeae053.png)

### Expected behavior:
The time spent on the two queries differs too much.

Because UUID is a tag field, so the first flux query ( is filter first then limit), and the second query (is limit first then filter) should no big difference.

So I guess the `contains` function does not use the group key for filtering, but scans all the data。

### Use Case:

Our team used influxdb-v2, but that is the bottleneck of our project now.

I have tried to use multiple `or` operations to replace `contains` function, but when the number of filters is large(70+), the or operation is slower.

Contributor guide

Open the contributing guide

Research direction

Start by running the two Flux queries in the issue and comparing their execution times with the same tag-filtering data. Investigate whether contains uses the UUID group key before scanning rows; done means the two query forms no longer show the reported performance gap, with a regression test covering the behavior.

Written by the indexing model from the issue text.

Assessment

Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.