huggingface / huggingface/llm-vscode

Give too many <MID> <PRE> <SUF> inline response when load custom LLM model with llm-vscode-server

Open
#104 4 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
TypeScript
Stars
1.3k
Forks
142
Avg merge
3m
Merged PRs (30d)
2

Description

## Environment
- api server project: https://github.com/wangcx18/llm-vscode-inference-server (based on `vllm`)
- model: Phind-CodeLlama-34B-v2-AWQ from https://huggingface.co/TheBloke/Phind-CodeLlama-34B-v2-AWQ
- plugin: llm-vscode 0.1.6, settings as below:
- "llm.tokenizer": {"path": "C:\\Users\\XXX\\tokenizer.json"} `(which is get from model folder)`
## Phenomenon
Then set the endpoint to `http://localhost:8000/generate` and plugins works, but given the and other symbols which make the code completions won't work well any more.
## Question
- Is something wrong that cause this error?
- Is there some exact tutorials that instructing developer to deploy a custom model that is totally available to work with llm-vscode?

Thanks for reading and thinking!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.