deepspeedai / deepspeedai/DeepSpeed

[REQUEST] Code sample to use DeepSpeed inference without having to run deepspeed cmd line in production set

Open
#1,702 4 comments 18 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

Is your feature request related to a problem? Please describe.
The current examples for DeepSpeed inference uses cmd line 'deepspeed' that internally uses launcher modules of deepspeed to initialize the NCCL/MPI backends and discovers the ranks, and world size etc. While using command line to run inference is good for the developer, but not useful for production use. Most of the production code uses serving stack to serve the inference real-time and uses python packages to initialize the frameworks like deepspeed and can't rely on running cmd line executions for each inference request.

Describe the solution you'd like
Provide a clear example of using deepspeed inference w/o using command line 'deepspeed'. Show in that example how to initialize the backend by using deepspeed.init_distributed() and call deepspeed.init_inference() calls and get rid of having to run the command line. I called deepspeed.init_distributed() in my code but the backend always fails to initilize even though I set all the right env variables for RANK, LOCAL_RANK, WORLD_SIZE etc.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start from the existing DeepSpeed inference examples and the deepspeed.init_distributed() and deepspeed.init_inference() entry points mentioned in the request. Add a production-oriented example that initializes the backend without the deepspeed command-line launcher, and verify that it explains the required environment variables and completes inference setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, documentation, machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.