influxdata / influxdata/influxdb
InfluxDB3 Core: DiskManager disabled prevents sort spilling, causes ResourcesExhausted on multi-day queries
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
## Problem
InfluxDB3 Core hardcodes `DiskManagerMode::Disabled` in the DataFusion `RuntimeEnv` configuration. This prevents the `ExternalSorter` from spilling to disk when the memory pool is exhausted.
The `DeduplicateExec` operator requires `Distribution::SinglePartition`, forcing a `SortPreservingMergeExec` to merge ALL file partitions into a single stream. With `max_parquet_fanout` greater than the file count, each file gets its own partition. For multi-day queries spanning 1000+ files, the merge operator holds 1000+ cursors simultaneously — all marked `can spill: false`.
This causes errors like:
```
ResourcesExhausted: Additional allocation failed with top memory consumers (bytes):
SortPreservingMergeExec(can spill: false): ...
```
...even with 100+ GB memory pools.
The `ExternalSorter` operators DO report `can spill: true`, but cannot actually spill because the `DiskManager` is disabled at the runtime level.
## Request
Make `DiskManager` configurable via CLI flag or environment variable (e.g., `--disk-manager-mode=auto` or `INFLUXDB3_DISK_MANAGER_MODE`). The default could remain disabled for backwards compatibility.
## Workaround
Reducing `max_parquet_fanout` below the file count forces DataFusion to use `target_partitions` (e.g., 80) instead of per-file partitions, reducing merge cursor count. However, this disables the pre-sorted file optimization and adds sort overhead.
## Environment
- InfluxDB3 Core (not Enterprise)
- DataFusion query execution
**Note:** Analysis performed using Claude Code based on source code review and production error patterns.
Contributor guide
Assessment
This issue has not been assessed yet.