DAMO-NLP-SG / DAMO-NLP-SG/LLM-R2
Question of LLM inference speed?
Open
- Dominant language
- Python
- Stars
- 54
- Forks
- 13
- PR merge metrics
- No merged PRs in 30d
Description
Hello, I noticed that the article mentions 5,000 queries, and the total time for rewriting them using a llm (taking TPC-H as an example) is only 3.40 seconds. This suggests that the inference time for each query is approximately 0.0006 seconds. Could you share which API you used to achieve such speed, or what kind of locally deployed model was utilized? Additionally, was the measurement conducted using techniques such as batching multiple queries for submission to the model or parallel processing with multithreading?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.