microsoft / microsoft/TransformerCompression
How to finetune with multi-gpus under data parallel setting?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 460
- Forks
- 58
- Avg merge
- 8h 40m
- Merged PRs (30d)
- 1
Description
Many thanks for sharing the amazing work!
I'm trying to finetune sliced 7b model on some large dataset with millions of samples.
But the distribute-model seems to be model parallel.
How can we finetune on the model with, let's say 8 gpus, under data parallel setting ?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the distribute-model entry point and any surrounding fine-tuning documentation to determine how its model-parallel behavior differs from the requested data-parallel setup. The issue does not name files or tests; completion would require an agreed, reproducible multi-GPU fine-tuning path or documentation explaining whether it is supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100