google / google/xls

Improve benchmarking infrastructure

Open
#1,226 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.9k
Forks
283
Avg merge
2d 10h
Merged PRs (30d)
135

Description

This is an umbrella issue for tracking progress and discussing tasks relevant to improving existing benchmarking infrastructure and providing new features that would improve the experience.

The objective here is to prepare a solution which would improve the process of developing DSLX design with awareness of physical design limitations. It would be good to have an easy way of defining flows which would gather a number of design metrics useful in design development. This could be done with a `dashboard rule` (https://github.com/google/xls/pull/1137) which runs various benchmarks and gathers the result in a form of html report.
Such solution would for sure be useful for example in the development of ZSTD codec (https://github.com/google/xls/issues/1211).

One of the most important performance metrics is the minimal clock period allowed for the design. However, obtaining this metric in the accurate variant requires sampling multiple `clock period` values for a given design through `Physical Design Flow` done by `OpenROAD` and checking if timings are met. For that, an integration with a parameter optimization framework `Vizier` (https://github.com/google/xls/pull/1160) would be a great addition.

In current form both efforts have a common component of executing various benchmarks and fetching performance metrics from them. Dashboard uses this for preparing HTML reports, while Vizier integration performs similar steps to evaluate suggested sets of parameter values.
I believe that this component should be extracted into a separate tool, as per suggestion from https://github.com/google/xls/pull/1160#issuecomment-1782917028, it should be usable outside of bazel, possibly something in the form of a `run_benchmark.py` from https://github.com/google/xls/issues/963.

Dashboard rule should be easy to extend with additional metrics as suggested in https://github.com/google/xls/pull/1137#issuecomment-1741107426 and https://github.com/google/xls/pull/1137#issuecomment-1825275638

One of the issues regarding `Physical Design` flow is the difference between implementation in `bazel_rules_hdl` and in `OpenROAD-Flow-Scripts` which is described in https://github.com/hdl/bazel_rules_hdl/issues/239.

The initial list of related issues and pull requests is presented below (includes inline links visible above):
* https://github.com/google/xls/issues/1058 & https://github.com/google/xls/pull/1137
* https://github.com/google/xls/pull/1160
* https://github.com/google/xls/issues/963
* external: Physical Design flow implementation: https://github.com/hdl/bazel_rules_hdl/issues/239
* https://github.com/google/xls/pull/1031
* https://github.com/google/xls/issues/1211
* https://github.com/google/xls/pull/1183
* https://github.com/google/xls/pull/964
* external: Python handling: https://github.com/hdl/bazel_rules_hdl/issues/209 & https://github.com/hdl/bazel_rules_hdl/pull/240
* https://github.com/hdl/bazel_rules_hdl/pull/247
* https://github.com/hdl/bazel_rules_hdl/pull/249
* https://github.com/hdl/bazel_rules_hdl/pull/250
* https://github.com/hdl/bazel_rules_hdl/pull/251
* https://github.com/hdl/bazel_rules_hdl/pull/252
* https://github.com/hdl/bazel_rules_hdl/pull/253
* https://github.com/hdl/bazel_rules_hdl/pull/254
* https://github.com/google/xls/pull/1230

CC @proppy

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.