microsoft / microsoft/onnxruntime
[Feature Request] Assess performance capability before a model is loaded
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
Assess performance capability without downloading the full model.
### Describe scenario use case
For some models, the performance may be a blocker. Since model downloads can be quite large, I wonder if there should be a way for web developers to know their machine performance class for running a model without downloading it completely first.
I believe this would involve running the model code with zeroed-out weights, which would still require buffer allocations but would allow the web app to catch out-of-memory errors or such. The model architecture would still needed to generate shaders, but this be much smaller than model weights.
cc @xenova @guschmue
Originally posted at https://github.com/xenova/transformers.js/pull/545#issuecomment-2147465443
Contributor guide
Research direction
Start by reviewing the linked Transformers.js discussion and the proposed zeroed-weight approach. Determine how model architecture, buffer allocation, shader generation, and out-of-memory detection should work before a full download. Done means a defined capability-assessment design with clear behavior for supported models and resource failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100