abetlen / abetlen/llama-cpp-python

Back End Grammar support for Web Server (separate from PR #855)

Open
#1,113 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
10.6k
Forks
1.4k
PR merge metrics
PR metrics pending

Description

**Is your feature request related to a problem? Please describe.**
It isn't really a problem, it is more a detriment to performance of the Server.

**Describe the solution you'd like**
Currently, you pass "grammar" as a function in your call, and then that grammar is parsed every single time for every single call you make to the web server.

I would like to pass the grammar to the webserver upon start up, and then use it for all calls thereafter.

**Describe alternatives you've considered**
The current version works, but every call looks like this:

```
from_string grammar:
root ::= answer
answer ::= [{] ws ]
ws ::= ws_10
string ::= ["] string_7 ["]
etc etc.

"POST /v1/completions HTTP/1.1" 200 OK
```

**Additional context**
There is likely a way for someone with higher level python skills than me to manipulate the code on the back end through a source build - I just figure, if I want it, I suspect others will want it at some point to....since grammar is quite frankly an astonishingly good addition.

Thanks.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.