pytorch / pytorch/benchmark

Torchbench models that don't run in dynamo runners

Open
#1,901 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1k
Forks
346
PR merge metrics
No merged PRs in 30d

Description

There's small nuances in how the dynamo runners benchmark models that can make certain torchbench models fail

Some models might be explicitly skipped, others might fail because of some dtype conversion. This can be frustrating because if you add a model to torchbench like clip or cm3leon you won't see it in the pt2dashboard so creating this giant tracker issue to solve this

To repro: look at logs in HUD for e.g https://ossci-raw-job-status.s3.amazonaws.com/log/16535270177 and compare to model names in models/ and canary_models/

If something is showing up in unique to torchbench that means it's not showing up in the pt2 dashboard

There are some concrete things we could do better in dynamo runners like starting with loudly erroring but should also track what these failures are

Notably I found this problem out after investigating stable diffusion and cm3leon

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the HUD log for the referenced job and compare its model names with models/ and canary_models/. Identify models listed as unique to torchbench, then trace whether explicit skips or dtype conversion failures prevent them from appearing in the pt2 dashboard. Done criteria are not defined because this is a broad tracker; the failure cases need to be separated into concrete fixes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.