huggingface / huggingface/optimum-nvidia
Qwen2.5-VL support
- Dominant language
- Python
- Stars
- 1k
- Forks
- 103
- Avg merge
- 2m
- Merged PRs (30d)
- 1
Description
Although Qwen2.5-VL is implemented in Transformers, it is not compatible with optimum-nvidia.
```bash
MODEL_NAME="Qwen/Qwen2.5-VL-3B-Instruct"
DESTINATION="./$MODEL_NAME"
HF_TOKEN=""
docker run \
--rm \
-it \
--gpus=1 \
--shm-size=1g \
-v "$DESTINATION":/engine \
-e HF_TOKEN=$HF_TOKEN \
-e HF_HUB_ENABLE_HF_TRANSFER=1 \
huggingface/optimum-nvidia:v0.1.0b9-py310 \
bash -c "pip install -U transformers && optimum-cli export trtllm \
--tp=1 \
--pp=1 \
--max-batch-size=64 \
--max-input-length 4096 \
--max-output-length 8192 \
--max-beams-width=1 \
--destination /tmp/engine \
$MODEL_NAME && cp -rL /tmp/engine/* /engine/"
```
[TensorRT-LLM] TensorRT-LLM version: 0.16.0
config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.37k/1.37k [00:00<00:00, 7.43MB/s]
Traceback (most recent call last):
File "/usr/local/bin/optimum-cli", line 8, in
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/dist-packages/optimum/commands/optimum_cli.py", line 208, in main
service.run()
File "/usr/local/lib/python3.12/dist-packages/optimum/nvidia/subpackage/commands/export.py", line 96, in run
model = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/huggingface_hub/hub_mixin.py", line 566, in from_pretrained
instance = cls._from_pretrained(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/optimum/nvidia/models/auto.py", line 72, in _from_pretrained
raise UnsupportedModelException(model_type)
optimum.nvidia.errors.UnsupportedModelException: Model of type qwen2_5_vl is not supported. Please open-up an issue at https://github.com/huggingface/optimum-nvidia/issues
Contributor guide
Assessment
This issue has not been assessed yet.