influxdata / influxdata/influxdb

Query memory limits not always observed

Open
#21,064 1 comment 3 reactions 0 assignees View on GitHub
area/2.x area/flux area/memory area/queries
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. Clean boot an influxdb server and note the steady state memory usage (on my instance this is ~2.5GB), with no active ingest or query workloads and with query thresholds set to constrain query memory usage to a modest amount:
```
query-concurrency: 1
query-initial-memory-bytes: 0
query-max-memory-bytes: 0
query-memory-bytes: 134217728
query-queue-size: 1
```
2. Initiate a single query which demands significant memory:
```
from(bucket: "hosts")
|> range(start: -90d, stop: now())
|> filter(fn: (r) => r["_measurement"] == "some_measurement")
|> aggregateWindow(every: 1ms, fn: mean, createEmpty: true)
```

__Expected behavior:__
Memory usage should not exceed the steady state plus the query-memory-bytes (in my case 2634217728 B) and query should fail with memory allocation error.

__Actual behavior:__
Memory is continually allocated until it exceeds the hosts resources (in my case a 5 GB container hard limit) which leads to a container reboot after the query fails to complete or error out.

__Environment info:__
* System info: Linux 4.19.0-10-amd64 x86_64
* InfluxDB version: InfluxDB 2.0.4 (git: 4e7a59bb9a) build_date: 2021-02-08T17:47:02Z

__Config:__
I have attempted a variety of configuration permutations to avoid this issue, all with the same result, so I am presenting the most simplistic that I believe should lead to the expected behaviour.
```
query-concurrency: 1
query-initial-memory-bytes: 0
query-max-memory-bytes: 0
query-memory-bytes: 134217728
query-queue-size: 1
```

Contributor guide

Open the contributing guide

Research direction

No source file or test is named. Start by reproducing the query with the listed memory settings and inspect the query memory-limit path; done means the query stays within configured query-memory-bytes and returns a memory allocation error instead of exhausting the host.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.