abetlen / abetlen/llama-cpp-python

Back End Grammar support for Web Server (separate from PR #855)

未关闭
#1,113 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
enhancement
主要语言
Python
星标
10.6k
派生
1.4k
PR 合并指标
PR 指标待抓取

描述

**Is your feature request related to a problem? Please describe.**
It isn't really a problem, it is more a detriment to performance of the Server.

**Describe the solution you'd like**
Currently, you pass "grammar" as a function in your call, and then that grammar is parsed every single time for every single call you make to the web server.

I would like to pass the grammar to the webserver upon start up, and then use it for all calls thereafter.

**Describe alternatives you've considered**
The current version works, but every call looks like this:

```
from_string grammar:
root ::= answer
answer ::= [{] ws ]
ws ::= ws_10
string ::= ["] string_7 ["]
etc etc.

"POST /v1/completions HTTP/1.1" 200 OK
```

**Additional context**
There is likely a way for someone with higher level python skills than me to manipulate the code on the back end through a source build - I just figure, if I want it, I suspect others will want it at some point to....since grammar is quite frankly an astonishingly good addition.

Thanks.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。