microsoft / microsoft/onnxruntime

[Feature Request] Cast Float16 model to Float32 [Web]

Open
#17,230 6 comments 1 reaction 0 assignees View on GitHub
feature request model:transformer platform:web
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

ORT-web can load float16 models and run it, but currently can't be operated the WebGPU provider. The same model, when casted to float32, can be accelerated with WebGPU for a massive speedup.

(I have verified this locally, but would need time to share a minimum implementation for comparison)

Implementing float16 for all operations is a lot of work, but is it possible to implement casting/'hydration' for the weights instead? So bandwith and storage can be reduced by 50% while also enabling WebGPU acceleration.

Model: Casted CLIP JS by rocca [clip-image-vit-32-float32.onnx](https://huggingface.co/rocca/openai-clip-js/blob/main/clip-image-vit-32-float32.onnx), and [details about casting that model to float32](https://github.com/josephrocca/openai-clip-js)
(CLIP is originally in float16)

Casting to FP16 by:

```python
# on python
import onnx
from onnxconverter_common import float16

model = onnx.load("clip-image-vit-32-float32.onnx")
model_fp16 = float16.convert_float_to_float16(model)
onnx.save(model_fp16, "clip-image-vit-32-float16.onnx")
```

Related: https://github.com/microsoft/onnxruntime/issues/9758

### Describe scenario use case

Smaller model to be downloaded and stored by users when using apps built by onnxruntime-web.

Contributor guide

Open the contributing guide

Research direction

The issue names ORT-web, the WebGPU provider, and related issue #9758 but no repository files or tests. Start by tracing the ORT-web model-loading path and reviewing the related issue, then define the supported weight-casting scope. Done should include a verified smaller float16 model that runs through WebGPU, using the CLIP model as the scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
machine-learning, performance, web-dev
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.