huggingface / huggingface/llm-vscode

Issue with Self-Hosted LLM Integration Using vLLM

Open
#146 8 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
TypeScript
Stars
1.3k
Forks
142
Avg merge
3m
Merged PRs (30d)
2

Description

Hi,

I’m trying to integrate a self-hosted LLM with the extension, which uses vLLM (supports the OpenAI API standard). I've configured the HTTP endpoint and model name, but I'm unsure where to include the API key.

I tried to add headers to the json settings, but I'm facing the following error when the code completion request is sent:
`serde json error: data did not match any variant of untagged enum OpenAIAPIResponse`

Could you guide me on the correct approach for passing the API key?

Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

No file, test, or entry point is identified in the issue. Start by tracing how the HTTP endpoint, model name, and request headers are configured for the vLLM OpenAI-compatible service, then reproduce the code-completion request and inspect the serde json error. Done means the API key can be passed through the supported configuration and the response is handled without the OpenAIAPIResponse parsing error.

Written by the indexing model from the issue text.

Assessment

Domain
ai, api, authentication
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.