iree-org / iree-org/iree-comparative-benchmark

[Migration] Update dashboard to process results from openxla-benchmark

Open
#39 5 comments 0 reactions 1 assignee Claimed by @beckerhe View on GitHub
Dominant language
Python
Stars
10
Forks
8
PR merge metrics
No merged PRs in 30d

Description

Benchmark results uploaded from openxla-benchmark are a little different from iree-samples.

Here is the example from T5_LARGE_FP32_JAX benchmarks:

From iree-samples: https://gist.github.com/pzread/2be3b2db7c0ffa14518085f08e33b815
From openxla-benchmarks: https://gist.github.com/pzread/cf3e089d0a4e0d5c452aebc1df6dbc44

The major differences are:

- Changes in benchmark id and benchmark name
- Changes in some metadata values (e.g. `data_type` and `device` field)
- Changes in the format of `python_environment` field

I think the pain point will be changing the benchmark id, which means we might need a backfill of the historical data. We can still show new data in the dashboard first then backfill the previous data

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.