NVIDIA / NVIDIA/TensorRT-Edge-LLM

Missing model_config argument in VLM visual ONNX export

Open Beginner friendly
#107 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
563
Forks
135
Avg merge
14h 13m
Merged PRs (30d)
1

Description

Describe the bug

It seems it is not possible to serve Qwen 3.5-0.8B (a natively Vision-language model) out of the box via the python high level API.

There seems to be missing a param in the python onnx export of the visual part of the model in the engine.py file.

I was able to correct and serve Qwen3.5-0.8B via teh high level API it in this way:

diff --git a/experimental/server/engine.py b/experimental/server/engine.py
index e768751..df7eb07 100644
--- a/experimental/server/engine.py
+++ b/experimental/server/engine.py
@@ -522,12 +522,14 @@ class LLM:
         import torch
 
         _ensure_export_package()
+        from tensorrt_edgellm.config import ModelConfig
         from tensorrt_edgellm.scripts.export import (_export_visual,
                                                      _load_all_weights,
                                                      _load_config)
 
         config = _load_config(self._model_dir)
         weights = _load_all_weights(self._model_dir)
+        model_config = ModelConfig.from_pretrained(self._model_dir)
         _export_visual(
             self._model_dir,
             self._visual_onnx_dir,
@@ -535,6 +537,7 @@ class LLM:
             config,
             self._model_type,
             torch.float16,
+            model_config=model_config,
         )
         logger.info(
             "Visual ONNX export complete: %s",

Let me know if you agree this is a bug or maybe i am missing something.
I can also open a PR with the fix after.

Steps/Code to reproduce bug

(venv) root@217da7dcd3f2:/workspace# python -m experimental.server --model Qwen/Qwen3.5-0.8B --port 8000
10:20:56 INFO edgellm.server: Resolving model: Qwen/Qwen3.5-0.8B
10:20:56 INFO edgellm.server: Downloading Qwen/Qwen3.5-0.8B from Hugging Face Hub ...
10:20:57 INFO httpx: HTTP Request: GET https://huggingface.co/api/models/Qwen/Qwen3.5-0.8B/revision/main "HTTP/1.1 200 OK"
Fetching 13 files: 100%|████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 14540.25it/s]
Download complete: : 0.00B [00:00, ?B/s] 10:20:57 INFO edgellm.server: Detected VLM model (type=qwen3_5)/13 [00:00<?, ?it/s]
10:20:57 INFO edgellm.server: Using cached ONNX: /root/.cache/huggingface/hub/models--Qwen--Qwen3.5-0.8B/snapshots/2fc06364715b967f1860aea9cf38778875588b17/.edgellm/onnx/llm
10:20:57 INFO edgellm.server: Exporting visual ONNX to /root/.cache/huggingface/hub/models--Qwen--Qwen3.5-0.8B/snapshots/2fc06364715b967f1860aea9cf38778875588b17/.edgellm/onnx/visual ...
Download complete: : 0.00B [00:00, ?B/s]
10:20:58 INFO tensorrt_edgellm.scripts.export: Loading shard: model.safetensors-00001-of-00001.safetensors
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/workspace/experimental/server/main.py", line 19, in
main()
File "/workspace/experimental/server/api_server.py", line 479, in main
llm = LLM(
^^^^
File "/workspace/experimental/server/engine.py", line 317, in init
self._init_from_model(
File "/workspace/experimental/server/engine.py", line 432, in _init_from_model
self._export_visual_onnx()
File "/workspace/experimental/server/engine.py", line 531, in _export_visual_onnx
_export_visual(
TypeError: _export_visual() missing 1 required positional argument: 'model_config'
(venv) root@217da7dcd3f2:/workspace#

Installation method:

Following teh instruction here, installing venv via pip https://nvidia.github.io/TensorRT-Edge-LLM/latest/user_guide/getting_started/installation.html#installation

Export command used:

python -m experimental.server --model Qwen/Qwen3.5-0.8B --port 8000
Expected behavior

Capable of Serving via the high level API a VLM model.

System information (x86 Host with GPU)

This is indipendent from the system.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in experimental/server/engine.py at _export_visual_onnx and reproduce with python -m experimental.server --model Qwen/Qwen3.5-0.8B --port 8000. Check the _export_visual call and the reported missing model_config argument; done means the visual ONNX export completes and the Qwen VLM serves through the high-level API.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.