huggingface / huggingface/llm-vscode

[Feature request] Adding a checker to see if a custom endpoint is working properly

Open
#106 1 comment 2 reactions 0 assignees View on GitHub
stale
Dominant language
TypeScript
Stars
1.3k
Forks
142
Avg merge
3m
Merged PRs (30d)
2

Description

I'm trying to run a model using the following command on my server:

```
docker run --gpus all --shm-size 1g -p 8080:80 -v /scratch/data:/data -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN -e HF_HUB_ENA^CE_HF_TRANSFER=0 ghcr.io/huggingface/text-generation-inference:1.1.0 --model-id bigcode/starcoder
```

But when I configure the IP in my editor: `http://XXXX:8080/generate` I would like to have a test from the editor that tell me whether or not the editor can successfully connect.

It could be helpful in the settings or as a dedicated command from the LLM extensions to verify that everything is in place and have helpful messages in case it is not.

As a check from the server side, I'm using `nvidia-smi` to see if additional GPU usage is happening

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the extension settings for the custom endpoint and the LLM extension command entry points; no specific files or tests are named in the issue. Define the connection check around the configured /generate endpoint, then verify that success is reported and connection failures produce helpful messages.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, typescript
Domain
api, developer-experience, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.