Supporting an Additional Input Modality (e.g. Video) in Optimum ExecuTorch

**Background**

Optimum ExecuTorch currently supports models with up to two input modalities (commonly text + image/text + audio).
While this covers many existing multimodal use cases, some newer multimodal models introduce three or more input modalities.

For example, models such as Qwen3-VL natively support:

* Text

* Image

* Video

At the moment, deploying such models on ExecuTorch requires either:

* Dropping one modality at export time, or

* Converting one modality (e.g. video) into another (e.g. sampled frames → image),

both of which reduce model completeness or expressiveness.

**This raises the question:**

Should Optimum ExecuTorch be extended to support an additional input modality, and if so, how should this be designed?

**Discussion Points**

I would like to open a discussion around the following aspects:

**API / Interface Design**

Should input modalities be treated as a fixed set (e.g. text/image/video)?

Or should ExecuTorch move toward a more generic “N-modality” input abstraction?

**Graph Representation & Export**

How should additional modality-specific preprocessing and operators be represented in the exported graph?

Are there existing constraints in ExecuTorch IR that assume two modalities?

**Runtime & Scheduling**

Would an additional modality require changes in scheduling, memory planning, or execution order?

Are there known limitations that would block such an extension?

**Incremental Support**

Would it be reasonable to start with a minimal third modality (e.g. video as a sequence of frames) and evolve from there?

**Motivation**

* Supporting more flexible multimodal inputs could:

* Improve compatibility with modern multimodal foundation models

* Reduce model-specific hacks or modality dropping

**Call for Feedback / Contributions**

I would appreciate feedback from maintainers and the community on:

* Whether this direction aligns with the project roadmap

* Potential design constraints or prior discussions

* Interest in collaborating on a prototype or RFC if this is deemed feasible

I’m happy to help with design exploration, implementation, or documentation if there is interest.