deepspeedai / deepspeedai/DeepSpeed
[REQUEST] Code sample to use DeepSpeed inference without having to run deepspeed cmd line in production set
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Is your feature request related to a problem? Please describe.
The current examples for DeepSpeed inference uses cmd line 'deepspeed' that internally uses launcher modules of deepspeed to initialize the NCCL/MPI backends and discovers the ranks, and world size etc. While using command line to run inference is good for the developer, but not useful for production use. Most of the production code uses serving stack to serve the inference real-time and uses python packages to initialize the frameworks like deepspeed and can't rely on running cmd line executions for each inference request.
Describe the solution you'd like
Provide a clear example of using deepspeed inference w/o using command line 'deepspeed'. Show in that example how to initialize the backend by using deepspeed.init_distributed() and call deepspeed.init_inference() calls and get rid of having to run the command line. I called deepspeed.init_distributed() in my code but the backend always fails to initilize even though I set all the right env variables for RANK, LOCAL_RANK, WORLD_SIZE etc.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start from the existing DeepSpeed inference examples and the deepspeed.init_distributed() and deepspeed.init_inference() entry points mentioned in the request. Add a production-oriented example that initializes the backend without the deepspeed command-line launcher, and verify that it explains the required environment variables and completes inference setup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, documentation, machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100