Integrate `mem_profile` utility to `bench.sh`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
[PR#16814](https://github.com/apache/datafusion/pull/16814) adds a new benchmark utility to retrieve memory statistics and print summary table.
We can run the binary directly with `cargo run --profile release-nonlto --bin mem_profile -- --bench-profile release-nonlto tpch --path benchmarks/data/tpch_sf1 --partitions 4 --format parquet --query 1`. However, there is still no integration with `bench.sh` to easily run individual benchmarks through mem_profile, nor is there a utility to compare results across different branches.
### Describe the solution you'd like
### Side Note
The way `mem_profile` collects the metrics and prints them out is quite different to other existing benchmark utilities.
For memory profiling, `mem_profile` spawns a new subprocess for each query execution. As a result, it does not generate a single output.json file for all bench queries like other benchmarks, but instead prints a summary table to stdout. To compare results across branches, we should either capture this stdout, or modify `mem_profile.rs` to also write results to a JSON file or other structured format.
### Steps
1. Navigate `bench.sh` and update places where it uses outdated entrypoint.
e.g. replace `--bin tpch` with `dfbench -- tpch`
(`mem_profile` passes subcommand and args to `dfbench`, so it would be easier to integrate it)
2. Add support for memory profiling mode in bench.sh
Modify bench.sh so that setting MEM_PROFILE=true runs each benchmark through the mem_profile binary instead of dfbench directly.
3. Extend compare.py and mem_profile.rs to allow side-by-side comparison of memory usage across branches or runs
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reviewing bench.sh and its current benchmark entry points, then inspect mem_profile.rs and compare.py. The work is complete when MEM_PROFILE runs benchmarks through mem_profile and memory results can be compared side by side across branches or runs using the chosen output format.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust, shell
- Domain
- performance, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100