abetlen / abetlen/llama-cpp-python

Back End Grammar support for Web Server (separate from PR #855)

オープン
#1,113 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
enhancement
主要言語
Python
スター
10.6k
フォーク
1.4k
PR マージ指標
PR 指標を取得中

説明

**Is your feature request related to a problem? Please describe.**
It isn't really a problem, it is more a detriment to performance of the Server.

**Describe the solution you'd like**
Currently, you pass "grammar" as a function in your call, and then that grammar is parsed every single time for every single call you make to the web server.

I would like to pass the grammar to the webserver upon start up, and then use it for all calls thereafter.

**Describe alternatives you've considered**
The current version works, but every call looks like this:

```
from_string grammar:
root ::= answer
answer ::= [{] ws ]
ws ::= ws_10
string ::= ["] string_7 ["]
etc etc.

"POST /v1/completions HTTP/1.1" 200 OK
```

**Additional context**
There is likely a way for someone with higher level python skills than me to manipulate the code on the back end through a source build - I just figure, if I want it, I suspect others will want it at some point to....since grammar is quite frankly an astonishingly good addition.

Thanks.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。