huggingface / huggingface/optimum-executorch

Add VLM export support (SmolVLM as starting point)

Open
#233 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
141
Forks
48
Avg merge
26m
Merged PRs (30d)
2

Description

Hey, I recently opened a PR in huggingface/transformers (#46360) adding vision/audio input support to the ExecuTorch export pipeline in executorch.py. The reviewer (zucchini-nlp) closed it and pointed me here, mentioning that multimodal ExecuTorch support is moving to optimum-executorch.

Looking at the repo, I can see text LLMs and image classifiers are both supported, but there's no export path for vision-language models, models that take image + text and generate text. SmolVLM seems like an obvious first target: it's a HF model, it's small enough to actually run on-device.

Before I start on anything, a few things I'd want to confirm:

Is VLM export support already being worked on somewhere, or is it an open gap?
Would the right shape be a new ExecuTorchModelForImageTextToText class following the ExecuTorchModelForCausalLM pattern?
Any known blockers on the ExecuTorch side for exporting a vision encoder + LLM decoder in one graph?

Happy to prototype something if this direction makes sense, just don't want to duplicate work.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.