Torchbench models that don't run in dynamo runners
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1k
- Forks
- 346
- PR merge metrics
- No merged PRs in 30d
Description
There's small nuances in how the dynamo runners benchmark models that can make certain torchbench models fail
Some models might be explicitly skipped, others might fail because of some dtype conversion. This can be frustrating because if you add a model to torchbench like clip or cm3leon you won't see it in the pt2dashboard so creating this giant tracker issue to solve this
To repro: look at logs in HUD for e.g https://ossci-raw-job-status.s3.amazonaws.com/log/16535270177 and compare to model names in models/ and canary_models/
If something is showing up in unique to torchbench that means it's not showing up in the pt2 dashboard
There are some concrete things we could do better in dynamo runners like starting with loudly erroring but should also track what these failures are
Notably I found this problem out after investigating stable diffusion and cm3leon
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the HUD log for the referenced job and compare its model names with models/ and canary_models/. Identify models listed as unique to torchbench, then trace whether explicit skips or dtype conversion failures prevent them from appearing in the pt2 dashboard. Done criteria are not defined because this is a broad tracker; the failure cases need to be separated into concrete fixes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100