huggingface / huggingface/llm-vscode

Running llama-cpp-python OpenAI compatible server

Open
#140 8 comments 3 reactions 0 assignees View on GitHub
stale
Dominant language
TypeScript
Stars
1.3k
Forks
142
Avg merge
3m
Merged PRs (30d)
2

Description

Requesting a little help here. Trying to test out copilot functionality with `llama-cpp-python` with this extension. Below is my configuration setting.

```bash
{
"[python]": {
"editor.formatOnType": true
},
"cmake.configureOnOpen": true,
"llm.backend": "openai",
"llm.configTemplate": "Custom",
"llm.url": "http://192.X.X.X:12080/v1/chat/completions",
"llm.fillInTheMiddle.enabled": false,
"llm.fillInTheMiddle.prefix": "

 ",

"llm.fillInTheMiddle.middle": " ",
"llm.fillInTheMiddle.suffix": " ",
"llm.requestBody": {
"parameters": {
"max_tokens": 60,
"temperature": 0.2,
"top_p": 0.95
}
},
"llm.contextWindow": 4096,
"llm.tokensToClear": [
""
],
"llm.tokenizer": null,
"llm.tlsSkipVerifyInsecure": true,
"llm.modelId": "",
}
```

I am seeing there is inference going on the server:

Screenshot 2024-04-23 at 11 10 01 PM

So I am not entirely sure what I am missing. Additionally I am trying to see the extension logs.. for the worker calls. But I don't see anything. Would you be able to give any guidance or some step by step explanation on how this can be done.

Thank you so much

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.