NVIDIA / NVIDIA/Megatron-LM

Are there plans to support multimodal MTP models in the future?

Open
#3,184 3 comments 0 reactions 1 assignee Claimed by @yashaswikarnati View on GitHub
community-request enhancement waiting-on-customer
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Tag the [@mcore-oncall](https://github.com/orgs/NVIDIA/teams/mcore-oncall)
to get oncall's attention to this issue.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

**“MTP currently does not support multimodal models.”**

```
self.megatron_model_meta = get_megatron_model_meta(args.hf_model_type)
self.visual = None
if args.mtp_num_layers:
raise ValueError('MTP currently does not support multimodal models.')
```
Are there plans to support multimodal MTP models in the future?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.