influxdata / influxdata/influxdb
Query memory limits not always observed
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. Clean boot an influxdb server and note the steady state memory usage (on my instance this is ~2.5GB), with no active ingest or query workloads and with query thresholds set to constrain query memory usage to a modest amount:
```
query-concurrency: 1
query-initial-memory-bytes: 0
query-max-memory-bytes: 0
query-memory-bytes: 134217728
query-queue-size: 1
```
2. Initiate a single query which demands significant memory:
```
from(bucket: "hosts")
|> range(start: -90d, stop: now())
|> filter(fn: (r) => r["_measurement"] == "some_measurement")
|> aggregateWindow(every: 1ms, fn: mean, createEmpty: true)
```
__Expected behavior:__
Memory usage should not exceed the steady state plus the query-memory-bytes (in my case 2634217728 B) and query should fail with memory allocation error.
__Actual behavior:__
Memory is continually allocated until it exceeds the hosts resources (in my case a 5 GB container hard limit) which leads to a container reboot after the query fails to complete or error out.
__Environment info:__
* System info: Linux 4.19.0-10-amd64 x86_64
* InfluxDB version: InfluxDB 2.0.4 (git: 4e7a59bb9a) build_date: 2021-02-08T17:47:02Z
__Config:__
I have attempted a variety of configuration permutations to avoid this issue, all with the same result, so I am presenting the most simplistic that I believe should lead to the expected behaviour.
```
query-concurrency: 1
query-initial-memory-bytes: 0
query-max-memory-bytes: 0
query-memory-bytes: 134217728
query-queue-size: 1
```
Contributor guide
Research direction
No source file or test is named. Start by reproducing the query with the listed memory settings and inspect the query memory-limit path; done means the query stays within configured query-memory-bytes and returns a memory allocation error instead of exhausting the host.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100