deepspeedai / deepspeedai/DeepSpeed

Can I call the forward function in parallel during the inference phase?

Open
#3,479 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug inference
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

I inited a model.
Can I call function model(args) in parallel (e.g. multiple requests) during inference ?
I got the error when qps >= 2

File "/home/web_server/src/processor.py", line 41, in process
res = self.model.predict(input)
File "/home/web_server/predict_en.py", line 379, in predict
res = forward_step_func(self.model, input)
File "/home/web_server/src/predict_en.py", line 89, in forward_step_func
output_tensor = model(input),
File "/home/web_server/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/web_server/anaconda3/lib/python3.8/site-packages/deepspeed/utils/nvtx.py", line 11, in wrapped_fn
return func(*args, **kwargs)
File "/home/web_server/anaconda3/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1668, in forward
self.start_timers(self.engine_timers.forward_timers)
File "/home/web_server/anaconda3/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2081, in start_timers
self.timers(name).start()
File "/home/web_server/anaconda3/lib/python3.8/site-packages/deepspeed/utils/timer.py", line 44, in start
assert not self.started
, f"{self.name
} timer has already been started"
AssertionError: forward_microstep timer has already been started

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with processor.py:41 and predict_en.py at predict and forward_step_func, then inspect DeepSpeed runtime/engine.py:1668 and utils/timer.py:44. Reproduce concurrent inference calls at QPS >= 2 and determine whether shared model state or the forward_microstep timer causes the assertion. Done means the supported concurrency behavior and the cause are established, with a regression test or documented limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.