Move functionality out of benchmark_main
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 283
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 135
Description
We should reduce the functionality of `benchmark_main`. It basically tries to replicate various stages of the build process to gather metrics. I think a less error prone process would be to gather metrics from build artifacts (schedule, block IR (which we don't yet dump), etc) and not actually run anything in benchmark main. This would avoid skew between build targets and what benchmark_main actually does. There is the issue of how to measure the time it takes to build the artifacts (optimization time, etc) but perhaps that might be solved by indicating how the build artifacts were generated perhaps as comments at the top of the artifact files, or maybe a separate artifact file.
One approach might be for the build rules to dump `.metrics` files with data for benchmark script to scrap. I guess the metrics file could also include the instructions on how to rebuild the artifact in order to time how long it takes.
Contributor guide
Assessment
This issue has not been assessed yet.