apache / apache/arrow

[C++] 'tpch' R benchmark results appear to have regressed on `test-mac-arm`, investigation needed

Open
#36,783 5 comments 0 reactions 0 assignees View on GitHub
Component: Benchmarking Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Let's look at https://conbench.ursa.dev/benchmark-results/064b8b034e6479cb80005f9cbb678ce5/.

This shows the historical evolution of the benchmark called `tpch` with case parameters `engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-21, scale_factor=1` on a machine that we call `test-mac-arm` running Darwin on arm64. Screenshot of results over time:

![image](https://github.com/apache/arrow/assets/265630/ffad13d4-028c-4c6e-8c77-96403a836b8b)

The time axis corresponds to code evolution on the default branch.

We can see that the measurement results changed from ~2 seconds to ~4 seconds in June, and now in the last few days there was another step from ~4 seconds to ~6 seconds.

I think it would be rather valuable to understand what happened here. Either there is a weakness in the experimental setup (limiting the value of this and similar benchmarks going forward from here, important to know) or there is indeed a significant and interesting "performance" change (also important to know).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.