abetlen / abetlen/llama-cpp-python

[Question] why i have 40 seconds more for the loading of the model, when i compare with llama-cpp

未關閉
#982 3 則留言 1 個 reaction 已指派 0 人 在 GitHub 檢視
bug
主要語言
Python
星號
10.6k
分支
1.4k
PR 合併指標
PR 指標待擷取

描述

Hello Guys,

I'm wondering about performence, which is very strange
on the same server, i ran the same model with query, and the loading time is totally differente between llama-cpp python and llama-cpp, in both case i put the same option (like number of threads ect...)

== lama-cpp -->
**llama_print_timings: load time = 2062.63 ms**
llama_print_timings: sample time = 36.35 ms / 55 runs ( 0.66 ms per token, 1513.15 tokens per second)
llama_print_timings: prompt eval time = 7583.04 ms / 97 tokens ( 78.18 ms per token, 12.79 tokens per second)
llama_print_timings: eval time = 8272.12 ms / 54 runs ( 153.19 ms per token, 6.53 tokens per second)
llama_print_timings: total time = 15922.60 ms

===== python -->

**llama_print_timings: load time = 42078.39 ms**
llama_print_timings: sample time = 92.67 ms / 142 runs ( 0.65 ms per token, 1532.30 tokens per second)
llama_print_timings: prompt eval time = 137100.44 ms / 1546 tokens ( 88.68 ms per token, 11.28 tokens per second)
llama_print_timings: eval time = 25390.80 ms / 141 runs ( 180.08 ms per token, 5.55 tokens per second)
llama_print_timings: total time = 163280.32 ms

so i really wondering why there such a huge difference.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。