Disproportionate memory use for `DISTINCT ON` query
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
We have a [parquet file](https://digitalsociety-public.fsn1.your-objectstorage.com/f94d0c87-8798-4bf6-9c98-8d89971e2539.parquet) (built from [public data](https://statistics.gov.scot/data/domestic-energy-performance-certificates)) with 106 columns and 1M rows which is 131.14 MiB in size (compressed, 913.89 MiB uncompressed).
When running a `DISTINCT ON` query using the unbounded memory pool, memory use climbs to over 160 GiB for this query:
```sql
SELECT DISTINCT
ON ("ADDRESS1", "ADDRESS2", "ADDRESS3", "POSTCODE") *
FROM
table
ORDER BY
"ADDRESS1",
"ADDRESS2",
"ADDRESS3",
"POSTCODE",
"INSPECTION_DATE" DESC
```
When using a fair spill pool with 10 GiB, memory usage reaches "only" 30 GiB.
These results were observed on my local machine (MacBook Pro). On a production machine with the same 10 GiB limit we have seen a graceful allocation failure:
```
Resources exhausted: Failed to allocate additional 55.0 MB for GroupedHashAggregateStream[3]
```
This makes me think it could be the same underlying issue as https://github.com/apache/datafusion/issues/13831, exacerbated by the many columns.
### To Reproduce
See the parquet file and SQL query in the description above.
### Expected behavior
In an ideal world, the memory usage for this query would respect the memory pool limit (or only use "small" allocations as described in [the docs](https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/trait.MemoryPool.html#memory-management-design)).
### Additional context
I'm happy to help diagnose this further (and potentially fix) with some advice on how to profile the memory use or narrow down the cause. For now I just wanted to capture the issue to see if it's known as I imagine it won't be an easy fix 😄
I know there are a few issues related to memory management floating around atm but none that I could see directly mentioned `DISTINCT ON`, so apologies if this is a duplicate.
Contributor guide
Research direction
Reproduce the SQL query against the linked parquet file, then read the DataFusion MemoryPool documentation and trace the allocation path named in the error, GroupedHashAggregateStream. The investigation is complete when the cause of the excess allocation is isolated and the query respects the configured memory-pool limit, with regression coverage if the relevant test location is found.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100