microsoft / microsoft/onnxruntime-genai

cannot build Qwen/Qwen3-14B on machine with 64GB RAM

Open
#2,047 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.1k
Forks
354
Avg merge
2d 16h
Merged PRs (30d)
85

Description

OS: Windows 11
onnxruntime-genai: 0.12.1
RAM: 64GB
`python -m onnxruntime_genai.models.builder -p int4 -e webgpu -m Qwen/Qwen3-14B -o Qwen3-14B`

It reports the following error
```
Reading decoder layer 37
Reading decoder layer 38
Reading decoder layer 39
Reading final norm
Reading LM head
Saving ONNX model in E:\ai-models\Qwen3-14B\onnx-webgpu
Traceback (most recent call last):
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnx_ir\serde.py", line 100, in wrapper
return func(*args, **kwargs)
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnx_ir\serde.py", line 1787, in serialize_tensor_into
tensor_proto.raw_data = from_.tobytes()
~~~~~~~~~~~~~^^
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnx_ir\_core.py", line 1078, in tobytes
return self._evaluate().tobytes()
~~~~~~~~~~~~~~~~~~~~~~~~^^
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnx_ir\tensor_adapters.py", line 206, in tobytes
return bytes(data)
MemoryError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnx_ir\serde.py", line 100, in wrapper
return func(*args, **kwargs)
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnx_ir\serde.py", line 1582, in serialize_graph_into
serialize_tensor_into(graph_proto.initializer.add(), from_=value.const_value)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnx_ir\serde.py", line 102, in wrapper
raise SerdeError(
f"Error calling {func.__name__} with: {arg_capturer(*args, **kwargs)}"
) from e
onnx_ir.serde.SerdeError: Error calling serialize_tensor_into with: LazyTensor(func=.tensor_func at 0x0000024E7CF10540>, name='lm_head.MatMul.weight')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnx_ir\serde.py", line 100, in wrapper
return func(*args, **kwargs)
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnx_ir\serde.py", line 1384, in serialize_model_into
serialize_graph_into(model_proto.graph, from_.graph)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnx_ir\serde.py", line 102, in wrapper
raise SerdeError(
f"Error calling {func.__name__} with: {arg_capturer(*args, **kwargs)}"
) from e
onnx_ir.serde.SerdeError: Error calling serialize_graph_into with: name=main_graph, doc_string=None, len(inputs)=82, len(initializers)=445, len(nodes)=774, len(outputs)=81, metadata_props={}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnxruntime_genai\models\builder.py", line 502, in
create_model(
~~~~~~~~~~~~^
args.model_name,
^^^^^^^^^^^^^^^^
...<5 lines>...
**extra_options,
^^^^^^^^^^^^^^^^
)
^
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\torch\utils\_contextlib.py", line 124, in decorate_context
return func(*args, **kwargs)
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnxruntime_genai\models\builder.py", line 346, in create_model
onnx_model.save_model(output_dir)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnxruntime_genai\models\builders\base.py", line 748, in save_model
model = self.to_int4()
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnxruntime_genai\models\builders\base.py", line 729, in to_int4
model=ir.to_proto(self.model),
~~~~~~~~~~~^^^^^^^^^^^^
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnx_ir\serde.py", line 277, in to_proto
return serialize_model(ir_object)
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnx_ir\serde.py", line 1356, in serialize_model
return serialize_model_into(onnx.ModelProto(), from_=model)
File "C:\Users\xiaofeihan\AppData\Roaming\Python\Python313\site-packages\onnx_ir\serde.py", line 102, in wrapper
raise SerdeError(
f"Error calling {func.__name__} with: {arg_capturer(*args, **kwargs)}"
) from e
onnx_ir.serde.SerdeError: Error calling serialize_model_into with: ir_version=10, producer_name=onnxruntime-genai, producer_version=None, domain=None,
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with onnxruntime_genai/models/builder.py:create_model and builders/base.py:save_model and to_int4, which appear in the traceback. Reproduce the provided Qwen3-14B int4 WebGPU command and inspect the serialization step; done means the export completes on a Windows 11 machine with 64GB RAM without MemoryError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.