microsoft / microsoft/onnxruntime
[Feature Request] Cast Float16 model to Float32 [Web]
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the feature request
ORT-web can load float16 models and run it, but currently can't be operated the WebGPU provider. The same model, when casted to float32, can be accelerated with WebGPU for a massive speedup.
(I have verified this locally, but would need time to share a minimum implementation for comparison)
Implementing float16 for all operations is a lot of work, but is it possible to implement casting/'hydration' for the weights instead? So bandwith and storage can be reduced by 50% while also enabling WebGPU acceleration.
Model: Casted CLIP JS by rocca [clip-image-vit-32-float32.onnx](https://huggingface.co/rocca/openai-clip-js/blob/main/clip-image-vit-32-float32.onnx), and [details about casting that model to float32](https://github.com/josephrocca/openai-clip-js)
(CLIP is originally in float16)
Casting to FP16 by:
```python
# on python
import onnx
from onnxconverter_common import float16
model = onnx.load("clip-image-vit-32-float32.onnx")
model_fp16 = float16.convert_float_to_float16(model)
onnx.save(model_fp16, "clip-image-vit-32-float16.onnx")
```
Related: https://github.com/microsoft/onnxruntime/issues/9758
### Describe scenario use case
Smaller model to be downloaded and stored by users when using apps built by onnxruntime-web.
Contributor guide
Research direction
The issue names ORT-web, the WebGPU provider, and related issue #9758 but no repository files or tests. Start by tracing the ORT-web model-loading path and reviewing the related issue, then define the supported weight-casting scope. Done should include a verified smaller float16 model that runs through WebGPU, using the CLIP model as the scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- machine-learning, performance, web-dev
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100