abetlen / abetlen/llama-cpp-python

Back End Grammar support for Web Server (separate from PR #855)

未關閉
#1,113 1 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
enhancement
主要語言
Python
星號
10.6k
分支
1.4k
PR 合併指標
PR 指標待擷取

描述

**Is your feature request related to a problem? Please describe.**
It isn't really a problem, it is more a detriment to performance of the Server.

**Describe the solution you'd like**
Currently, you pass "grammar" as a function in your call, and then that grammar is parsed every single time for every single call you make to the web server.

I would like to pass the grammar to the webserver upon start up, and then use it for all calls thereafter.

**Describe alternatives you've considered**
The current version works, but every call looks like this:

```
from_string grammar:
root ::= answer
answer ::= [{] ws ]
ws ::= ws_10
string ::= ["] string_7 ["]
etc etc.

"POST /v1/completions HTTP/1.1" 200 OK
```

**Additional context**
There is likely a way for someone with higher level python skills than me to manipulate the code on the back end through a source build - I just figure, if I want it, I suspect others will want it at some point to....since grammar is quite frankly an astonishingly good addition.

Thanks.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。