`generate_series` doesn't respect memory limit
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
You can trivial cause datafusion to use any amount of memory by simply running
```sql
select generate_series(9876543210);
```
Memory management functionality, e.g. [`MemoryPool`](https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/trait.MemoryPool.html) does't seem to have any effect
### To Reproduce
Run `datafusion-cli` with a memory limit, then run `generate_series`:
```bash
datafusion-cli -m 1g -c 'select generate_series(9876543210);'
```
Memory immediately jumps to ~20GB. (note this is not limited to `datafusion-cli`)
This query also hangs indefinitely, but in production we see posts being killed OOM for queries like this.
### Expected behavior
`generate_series` should either be streamed so it uses very little memory, or should be killed/constrained by the memory pool.
### Additional context
Same presumably applies to the `range` UDF.
cc @davidhewitt
Contributor guide
Research direction
Start by reproducing the issue with datafusion-cli using the provided generate_series query and a 1g memory limit. Trace the generate_series UDF and its interaction with MemoryPool, then compare the behavior with the range UDF mentioned in the report. Done means the query is streamed or constrained by the memory pool instead of exhausting memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100