microsoft / microsoft/onnxruntime

[Feature Request] Assess performance capability before a model is loaded

Open
#20,998 7 comments 6 reactions 0 assignees View on GitHub
ep:WebGPU feature request platform:web
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

Assess performance capability without downloading the full model.

### Describe scenario use case

For some models, the performance may be a blocker. Since model downloads can be quite large, I wonder if there should be a way for web developers to know their machine performance class for running a model without downloading it completely first.

I believe this would involve running the model code with zeroed-out weights, which would still require buffer allocations but would allow the web app to catch out-of-memory errors or such. The model architecture would still needed to generate shaders, but this be much smaller than model weights.

cc @xenova @guschmue

Originally posted at https://github.com/xenova/transformers.js/pull/545#issuecomment-2147465443

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the linked Transformers.js discussion and the proposed zeroed-weight approach. Determine how model architecture, buffer allocation, shader generation, and out-of-memory detection should work before a full download. Done means a defined capability-assessment design with clear behavior for supported models and resource failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.