apache / apache/datafusion

`generate_series` doesn't respect memory limit

Open
#12,919 5 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

You can trivial cause datafusion to use any amount of memory by simply running

```sql
select generate_series(9876543210);
```

Memory management functionality, e.g. [`MemoryPool`](https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/trait.MemoryPool.html) does't seem to have any effect

### To Reproduce

Run `datafusion-cli` with a memory limit, then run `generate_series`:

```bash
datafusion-cli -m 1g -c 'select generate_series(9876543210);'
```

Memory immediately jumps to ~20GB. (note this is not limited to `datafusion-cli`)

This query also hangs indefinitely, but in production we see posts being killed OOM for queries like this.

### Expected behavior

`generate_series` should either be streamed so it uses very little memory, or should be killed/constrained by the memory pool.

### Additional context

Same presumably applies to the `range` UDF.

cc @davidhewitt

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with datafusion-cli using the provided generate_series query and a 1g memory limit. Trace the generate_series UDF and its interaction with MemoryPool, then compare the behavior with the range UDF mentioned in the report. Done means the query is streamed or constrained by the memory pool instead of exhausting memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.