deepspeedai / deepspeedai/DeepSpeed
[QUESTION] How to do inference during deepspeed training
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
I'm willing to do several inference operations for validation during the distributed training processes. The inference operations invlove some functions of my model, which cannot be called when my model is wrapped by deepspeed.engine. What's the right way to do?
For example, I'm willing to call generate() of models in huggingface transformers. May I just use engine.module.generate() ?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the issue's description of DeepSpeed engine wrapping and the referenced engine.module.generate() entry point. Confirm the supported way to invoke model functions during distributed training, then document the guidance and validation constraints; no specific files or tests are named in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100