apache / apache/datafusion

Benchmarks should run with a configured memory limit

Open
#22,533 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

This would exercise the memory accounting feature, it would trigger operators to spill etc.
See this comment where the peak memory for running a query is 10GiB but some operators spill even with a memory limit of 50 GiB: https://github.com/apache/datafusion/issues/22526#issuecomment-4568611822

### Describe the solution you'd like
Instead of no memory limit, benchmarks should run with a default memory limit.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

No specific file or test is named. Start by locating the benchmark entry point and the existing memory-accounting configuration, then determine how benchmarks currently run without a limit. Done means benchmarks use a default memory limit and exercise accounting behavior such as operator spilling.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases, performance, testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.