influxdata / influxdata/influxdb

InfluxDB3 Core: DiskManager disabled prevents sort spilling, causes ResourcesExhausted on multi-day queries

Open
#27,328 3 comments 1 reaction 1 assignee Claimed by @alamb View on GitHub
v3 v3/core
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

## Problem

InfluxDB3 Core hardcodes `DiskManagerMode::Disabled` in the DataFusion `RuntimeEnv` configuration. This prevents the `ExternalSorter` from spilling to disk when the memory pool is exhausted.

The `DeduplicateExec` operator requires `Distribution::SinglePartition`, forcing a `SortPreservingMergeExec` to merge ALL file partitions into a single stream. With `max_parquet_fanout` greater than the file count, each file gets its own partition. For multi-day queries spanning 1000+ files, the merge operator holds 1000+ cursors simultaneously — all marked `can spill: false`.

This causes errors like:

```
ResourcesExhausted: Additional allocation failed with top memory consumers (bytes):
SortPreservingMergeExec(can spill: false): ...
```

...even with 100+ GB memory pools.

The `ExternalSorter` operators DO report `can spill: true`, but cannot actually spill because the `DiskManager` is disabled at the runtime level.

## Request

Make `DiskManager` configurable via CLI flag or environment variable (e.g., `--disk-manager-mode=auto` or `INFLUXDB3_DISK_MANAGER_MODE`). The default could remain disabled for backwards compatibility.

## Workaround

Reducing `max_parquet_fanout` below the file count forces DataFusion to use `target_partitions` (e.g., 80) instead of per-file partitions, reducing merge cursor count. However, this disables the pre-sorted file optimization and adds sort overhead.

## Environment

- InfluxDB3 Core (not Enterprise)
- DataFusion query execution

**Note:** Analysis performed using Claude Code based on source code review and production error patterns.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.