huggingface / huggingface/llm-vscode
[Feature request] Adding a checker to see if a custom endpoint is working properly
- Dominant language
- TypeScript
- Stars
- 1.3k
- Forks
- 142
- Avg merge
- 3m
- Merged PRs (30d)
- 2
Description
I'm trying to run a model using the following command on my server:
```
docker run --gpus all --shm-size 1g -p 8080:80 -v /scratch/data:/data -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN -e HF_HUB_ENA^CE_HF_TRANSFER=0 ghcr.io/huggingface/text-generation-inference:1.1.0 --model-id bigcode/starcoder
```
But when I configure the IP in my editor: `http://XXXX:8080/generate` I would like to have a test from the editor that tell me whether or not the editor can successfully connect.
It could be helpful in the settings or as a dedicated command from the LLM extensions to verify that everything is in place and have helpful messages in case it is not.
As a check from the server side, I'm using `nvidia-smi` to see if additional GPU usage is happening
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the extension settings for the custom endpoint and the LLM extension command entry points; no specific files or tests are named in the issue. Define the connection check around the configured /generate endpoint, then verify that success is reported and connection failures produce helpful messages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, typescript
- Domain
- api, developer-experience, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100