Memory held on in `datafusion-cli` (reported memory size increases over time)
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
When running several memory consuming queries in datafusion-cli, the system memory won't be released after the query ends, the memory usage from different queries are accumulated, until `datafusion-cli` is exited.
### To Reproduce
1. Compile and run datafusion-cli (commit 8c42a65a4)
2. I'm using MacOS's `Application Monitor` GUI to check memory usage of `datafusion-cli` process
3. Execute several memory-intensive queries, and memory usage accumulates over time.
```
select *
from generate_series(1,100000000) as t1(v1)
order by v1;
-- datafusion-cli process: 1.5G memory
select *
from generate_series(1,200000000) as t1(v1)
order by v1;
-- datafusion-cli process: 3.2G memory
select 1;
-- Still 3.2G
```
### Expected behavior
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Compile and run datafusion-cli at commit 8c42a65a4, then reproduce the two memory-intensive generate_series queries followed by SELECT 1 while monitoring the process in macOS Application Monitor. Trace the datafusion-cli query lifecycle to determine why memory remains allocated after each query; done means repeated queries no longer cause memory usage to accumulate after completion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100