modelscope / modelscope/ms-swift
Do you have plans on adding megatron support for contrastive learning?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Describe the feature
So as the titles says, I was very excited to train big foundation MoE embedding model using ms-swift framework, as it has both infonce loss support and expert parallel megatron support. Only later i found out that those two things are not compatible. Is there any chance you can add support for task_type = embeddings and infonce loss for megatron?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the task_type=embeddings, InfoNCE loss, and Megatron expert-parallel training paths in the repository, then trace where their compatibility is checked or limited. Compare the existing embedding and Megatron flows and define tests that demonstrate embeddings with InfoNCE running under Megatron expert parallelism; done means that combination is supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100