[C++] 'tpch' R benchmark results appear to have regressed on `test-mac-arm`, investigation needed
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Let's look at https://conbench.ursa.dev/benchmark-results/064b8b034e6479cb80005f9cbb678ce5/.
This shows the historical evolution of the benchmark called `tpch` with case parameters `engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-21, scale_factor=1` on a machine that we call `test-mac-arm` running Darwin on arm64. Screenshot of results over time:

The time axis corresponds to code evolution on the default branch.
We can see that the measurement results changed from ~2 seconds to ~4 seconds in June, and now in the last few days there was another step from ~4 seconds to ~6 seconds.
I think it would be rather valuable to understand what happened here. Either there is a weakness in the experimental setup (limiting the value of this and similar benchmarks going forward from here, important to know) or there is indeed a significant and interesting "performance" change (also important to know).
Contributor guide
Assessment
This issue has not been assessed yet.