microsoft / microsoft/MInference
[Question]: Why is running MInference/examples/run_vllm.py not as fast as running vllm alone?
Open
@iofu728 is already working on this.
Since Jul 16, 2024.
question
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 82
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 1
Description
Describe the issue
from vllm import LLM, SamplingParams
from minference import MInference
prompts = [
"Hello, my name is",
"The president of the United States is",
"The capital of France is",
"The future of AI is",
]
prompts = prompts*100
sampling_params = SamplingParams(
temperature=0.8,
top_p=0.95,
max_tokens=10,
)
model_name = "Qwen/Qwen2-7B-Instruct/"
llm = LLM(
model_name,
max_num_seqs=1,
enforce_eager=True,
)
# Patch MInference Module
minference_patch = MInference("vllm", model_name)
llm = minference_patch(llm)
outputs = llm.generate(prompts, sampling_params)
import time
t1 = time.time()
# Print the outputs.
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
t2 = time.time()
print('minference_time:',t2-t1)
print('=============================================================================')
from vllm import LLM, SamplingParams
from minference import MInference
prompts = [
"Hello, my name is",
"The president of the United States is",
"The capital of France is",
"The future of AI is",
]
prompts = prompts*100
sampling_params = SamplingParams(
temperature=0.8,
top_p=0.95,
max_tokens=10,
)
model_name = "Qwen/Qwen2-7B-Instruct/"
llm = LLM(
model_name,
max_num_seqs=1,
enforce_eager=True,
)
# Patch MInference Module
#minference_patch = MInference("vllm", model_name)
#llm = minference_patch(llm)
outputs = llm.generate(prompts, sampling_params)
import time
t1 = time.time()
# Print the outputs.
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
t2 = time.time()
print('vllm_time:',t2-t1)
result:
minference_time:0.0003895759582519531s
vllm_time:0.0002791881561279297s
Why is minference_time greater than vllm_time?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.