DAMO-NLP-SG / DAMO-NLP-SG/LLM-R2

Question of LLM inference speed?

Open
#8 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
54
Forks
13
PR merge metrics
No merged PRs in 30d

Description

Hello, I noticed that the article mentions 5,000 queries, and the total time for rewriting them using a llm (taking TPC-H as an example) is only 3.40 seconds. This suggests that the inference time for each query is approximately 0.0006 seconds. Could you share which API you used to achieve such speed, or what kind of locally deployed model was utilized? Additionally, was the measurement conducted using techniques such as batching multiple queries for submission to the model or parallel processing with multithreading?

Image

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.