Improve benchmarking infrastructure
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 283
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 135
Description
This is an umbrella issue for tracking progress and discussing tasks relevant to improving existing benchmarking infrastructure and providing new features that would improve the experience.
The objective here is to prepare a solution which would improve the process of developing DSLX design with awareness of physical design limitations. It would be good to have an easy way of defining flows which would gather a number of design metrics useful in design development. This could be done with a `dashboard rule` (https://github.com/google/xls/pull/1137) which runs various benchmarks and gathers the result in a form of html report.
Such solution would for sure be useful for example in the development of ZSTD codec (https://github.com/google/xls/issues/1211).
One of the most important performance metrics is the minimal clock period allowed for the design. However, obtaining this metric in the accurate variant requires sampling multiple `clock period` values for a given design through `Physical Design Flow` done by `OpenROAD` and checking if timings are met. For that, an integration with a parameter optimization framework `Vizier` (https://github.com/google/xls/pull/1160) would be a great addition.
In current form both efforts have a common component of executing various benchmarks and fetching performance metrics from them. Dashboard uses this for preparing HTML reports, while Vizier integration performs similar steps to evaluate suggested sets of parameter values.
I believe that this component should be extracted into a separate tool, as per suggestion from https://github.com/google/xls/pull/1160#issuecomment-1782917028, it should be usable outside of bazel, possibly something in the form of a `run_benchmark.py` from https://github.com/google/xls/issues/963.
Dashboard rule should be easy to extend with additional metrics as suggested in https://github.com/google/xls/pull/1137#issuecomment-1741107426 and https://github.com/google/xls/pull/1137#issuecomment-1825275638
One of the issues regarding `Physical Design` flow is the difference between implementation in `bazel_rules_hdl` and in `OpenROAD-Flow-Scripts` which is described in https://github.com/hdl/bazel_rules_hdl/issues/239.
The initial list of related issues and pull requests is presented below (includes inline links visible above):
* https://github.com/google/xls/issues/1058 & https://github.com/google/xls/pull/1137
* https://github.com/google/xls/pull/1160
* https://github.com/google/xls/issues/963
* external: Physical Design flow implementation: https://github.com/hdl/bazel_rules_hdl/issues/239
* https://github.com/google/xls/pull/1031
* https://github.com/google/xls/issues/1211
* https://github.com/google/xls/pull/1183
* https://github.com/google/xls/pull/964
* external: Python handling: https://github.com/hdl/bazel_rules_hdl/issues/209 & https://github.com/hdl/bazel_rules_hdl/pull/240
* https://github.com/hdl/bazel_rules_hdl/pull/247
* https://github.com/hdl/bazel_rules_hdl/pull/249
* https://github.com/hdl/bazel_rules_hdl/pull/250
* https://github.com/hdl/bazel_rules_hdl/pull/251
* https://github.com/hdl/bazel_rules_hdl/pull/252
* https://github.com/hdl/bazel_rules_hdl/pull/253
* https://github.com/hdl/bazel_rules_hdl/pull/254
* https://github.com/google/xls/pull/1230
CC @proppy
Contributor guide
Assessment
This issue has not been assessed yet.