apache / apache/datafusion

Integrate `mem_profile` utility to `bench.sh`

Open
#16,938 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

[PR#16814](https://github.com/apache/datafusion/pull/16814) adds a new benchmark utility to retrieve memory statistics and print summary table.

We can run the binary directly with `cargo run --profile release-nonlto --bin mem_profile -- --bench-profile release-nonlto tpch --path benchmarks/data/tpch_sf1 --partitions 4 --format parquet --query 1`. However, there is still no integration with `bench.sh` to easily run individual benchmarks through mem_profile, nor is there a utility to compare results across different branches.

### Describe the solution you'd like

### Side Note
The way `mem_profile` collects the metrics and prints them out is quite different to other existing benchmark utilities.
For memory profiling, `mem_profile` spawns a new subprocess for each query execution. As a result, it does not generate a single output.json file for all bench queries like other benchmarks, but instead prints a summary table to stdout. To compare results across branches, we should either capture this stdout, or modify `mem_profile.rs` to also write results to a JSON file or other structured format.

### Steps
1. Navigate `bench.sh` and update places where it uses outdated entrypoint.
e.g. replace `--bin tpch` with `dfbench -- tpch`
(`mem_profile` passes subcommand and args to `dfbench`, so it would be easier to integrate it)
2. Add support for memory profiling mode in bench.sh
Modify bench.sh so that setting MEM_PROFILE=true runs each benchmark through the mem_profile binary instead of dfbench directly.
3. Extend compare.py and mem_profile.rs to allow side-by-side comparison of memory usage across branches or runs

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing bench.sh and its current benchmark entry points, then inspect mem_profile.rs and compare.py. The work is complete when MEM_PROFILE runs benchmarks through mem_profile and memory results can be compared side by side across branches or runs using the chosen output format.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust, shell
Domain
performance, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.