[CI] Same-batch comparability for `/overview` data / `/overview` 数据的同批次可比性

Open
#2,304 0 comments 0 reactions 1 assignee View on GitHub

@edwingao28 is already working on this.

Since Jul 22, 2026.

Assessment

This issue has not been assessed yet.

Description

/overview data gaps aren't a timeout problem — normal benchmarks run ~15–80 min against a 480–500 min cap. The gap is that hardware isn't run in the same batch, so cross-hardware numbers aren't strictly comparable.

Not the fix: raising CI runtime/timeout.

Approach — a calibration workflow: pin one snapshot (same commit / image / model / config / workload), run it across the exec-relevant hardware, and publish the result as a single dated snapshot. Pilot small first; expand only if the pilot proves comparability. Pilot scope TBD in thread.

Next phase for https://github.com/SemiAnalysisAI/InferenceX-app/pull/611 — not a blocker for the current /overview PR.

/overview 的数据缺口不是超时问题——常规 benchmark 只跑约 15–80 分钟，而上限是 480–500 分钟。真正的缺口在于不同硬件没有在同一批次运行，因此跨硬件的数字并不严格可比。

不是解决方案：提高 CI 运行时长／超时上限。

方案——校准 workflow：固定一个 snapshot（相同的 commit／镜像／模型／配置／workload），在面向高层的相关硬件上运行，并将结果作为一个带日期的 snapshot 发布。先做小规模 pilot；仅当 pilot 证明可比性后再扩展。Pilot 范围在 issue 讨论中再定。

https://github.com/SemiAnalysisAI/InferenceX-app/pull/611 的下一阶段——不阻塞当前 /overview PR。

Dominant language
Python
Stars
1.7k
Forks
303
Avg merge
1d 13h
Merged PRs (30d)
284

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from SemiAnalysisAI/InferenceX

All issues in SemiAnalysisAI/InferenceX

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.