huggingface / huggingface/llm-ls

ollama backend does not support api keys

Open
#106 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
881
Forks
71
Avg merge
33m
Merged PRs (30d)
1

Description

Hi!

I'm trying to set up llm-ls via llm.nvim plugin and I'm hitting weird serdes errors. I set the following config:

```
{
backend="openai",
url = "http://192.168.0.61:8080/api",
api_token = "sk-xxxx", -- replace after publishing outside local network

lsp = {
cmd_env = { LLM_LOG_LEVEL = "DEBUG" }
},
}
```

Weird thing is, completion seems to work just fine. But after each completion, I get a serdes error per following log. Any idea what am I doing wrong? I find it hard to find docs on using open webui as a backend, as well as the open webui docs for their API (I believe it is supposed to be openai compatible, hence I set it so).

Relevant logs (api key stripped):
```
{"timestamp":"2024-09-18T20:44:02.195454Z","level":"INFO","message":"file:///home/krakonos/test.cpp changed","target":"llm_ls","line_number":682}
{"timestamp":"2024-09-18T20:44:02.195645Z","level":"DEBUG","message":"client asked to cancel request 4, but no such pending request exists, ignoring","target":"tower_lsp::service::pending","line_number":64}
{"timestamp":"2024-09-18T20:44:02.346897Z","level":"INFO","message":"received completion request","document_url":"file:///home/krakonos/test.cpp","cursor_line":"19","cursor_character":"49","language_id":"cpp","model":"deepseek-coder-v2:16b-lite-base-q4_0","backend":"OpenAi { url: \"http://192.168.0.61:8080/api\" }","ide":"neovim","request_body":"{\"temperature\":0.2,\"top_p\":0.95}","disable_url_path_completion":false,"target":"llm_ls","line_number":514,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.347146Z","level":"INFO","message":"completion type: SingleLine","completion_type":"single_line","target":"llm_ls","line_number":537,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.347201Z","level":"INFO","message":"built prompt in 0 ms","prompt":"<|fim▁begin|>#include \n#include \n\n\n\nclass A {\n\n};\n\nclass B {\n\n};\n\nint foo() {\n\n}\n\nint main() {\n std::shared_ptr a = std::make_shared();\n std::shared_ptr b = std::make_shared();<|fim▁hole|>\n return 0;\n}\n<|fim▁end|>","build_prompt_ms":"0","target":"llm_ls","line_number":204,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.347232Z","level":"INFO","message":"sending request to backend","headers":"{\"user-agent\": \"llm-ls/0.5.2; rust/unknown; ide/Neovim\", \"authorization\": \"Bearer sk-xxxx\"}","url":"http://192.168.0.61:8080/api/v1/completions","target":"llm_ls","line_number":254,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.347281Z","level":"DEBUG","message":"sending request to backend","headers":"{\"user-agent\": \"llm-ls/0.5.2; rust/unknown; ide/Neovim\", \"authorization\": \"Bearer sk-xxxx\"}","body":"{\"model\": String(\"deepseek-coder-v2:16b-lite-base-q4_0\"), \"prompt\": String(\"<|fim▁begin|>#include \\n#include \\n\\n\\n\\nclass A {\\n\\n};\\n\\nclass B {\\n\\n};\\n\\nint foo() {\\n\\n}\\n\\nint main() {\\n std::shared_ptr
a = std::make_shared();\\n std::shared_ptr b = std::make_shared();<|fim▁hole|>\\n return 0;\\n}\\n<|fim▁end|>\"), \"stream\": Bool(false), \"temperature\": Number(0.2), \"top_p\": Number(0.95)}","url":"http://192.168.0.61:8080/api/v1/completions","target":"llm_ls","line_number":255,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.347408Z","level":"DEBUG","message":"starting new connection: http://192.168.0.61:8080/","log.target":"reqwest::connect","log.module_path":"reqwest::connect","log.file":"/home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.20/src/connect.rs","log.line":429,"target":"reqwest::connect","line_number":429,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.347446Z","level":"DEBUG","message":"connecting to 192.168.0.61:8080","target":"hyper::client::connect::http","line_number":537,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.363434Z","level":"DEBUG","message":"connected to 192.168.0.61:8080","target":"hyper::client::connect::http","line_number":540,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.363816Z","level":"DEBUG","message":"flushed 637 bytes","target":"hyper::proto::h1::io","line_number":342}
{"timestamp":"2024-09-18T20:44:02.368285Z","level":"DEBUG","message":"parsed 5 headers","target":"hyper::proto::h1::io","line_number":207}
{"timestamp":"2024-09-18T20:44:02.368420Z","level":"DEBUG","message":"incoming body is content-length (22 bytes)","target":"hyper::proto::h1::conn","line_number":222}
{"timestamp":"2024-09-18T20:44:02.368492Z","level":"DEBUG","message":"incoming body completed","target":"hyper::proto::h1::conn","line_number":298}
{"timestamp":"2024-09-18T20:44:02.368619Z","level":"DEBUG","message":"pooling idle connection for (\"http\", 192.168.0.61:8080)","target":"hyper::client::pool","line_number":376,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.368860Z","level":"ERROR","err_msg":"serde json error: data did not match any variant of untagged enum OpenAIAPIResponse","target":"llm_ls::error","line_number":8,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:03.380607Z","level":"INFO","message":"file:///home/krakonos/test.cpp changed","target":"llm_ls","line_number":682}
{"timestamp":"2024-09-18T20:44:03.381212Z","level":"DEBUG","message":"client asked to cancel request 5, but no such pending request exists, ignoring","target":"tower_lsp::service::pending","line_number":64}
{"timestamp":"2024-09-18T20:44:03.531812Z","level":"INFO","message":"received completion request","document_url":"file:///home/krakonos/test.cpp","cursor_line":"20","cursor_character":"4","language_id":"cpp","model":"deepseek-coder-v2:16b-lite-base-q4_0","backend":"OpenAi { url: \"http://192.168.0.61:8080/api\" }","ide":"neovim","request_body":"{\"temperature\":0.2,\"top_p\":0.95}","disable_url_path_completion":false,"target":"llm_ls","line_number":514,"spans":[{"request_id":"e26d9290-1619-4990-9df8-b8b665f7621d","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:03.531954Z","level":"INFO","message":"completion type: SingleLine","completion_type":"single_line","target":"llm_ls","line_number":537,"spans":[{"request_id":"e26d9290-1619-4990-9df8-b8b665f7621d","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:03.532024Z","level":"INFO","message":"built prompt in 0 ms","prompt":"<|fim▁begin|>#include \n#include \n\n\n\nclass A {\n\n};\n\nclass B {\n\n};\n\nint foo() {\n\n}\n\nint main() {\n std::shared_ptr
a = std::make_shared();\n std::shared_ptr b = std::make_shared();\n <|fim▁hole|>\n return 0;\n}\n<|fim▁end|>","build_prompt_ms":"0","target":"llm_ls","line_number":204,"spans":[{"request_id":"e26d9290-1619-4990-9df8-b8b665f7621d","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:03.532053Z","level":"INFO","message":"sending request to backend","headers":"{\"user-agent\": \"llm-ls/0.5.2; rust/unknown; ide/Neovim\", \"authorization\": \"Bearer sk-xxxx\"}","url":"http://192.168.0.61:8080/api/v1/completions","target":"llm_ls","line_number":254,"spans":[{"request_id":"e26d9290-1619-4990-9df8-b8b665f7621d","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:03.532073Z","level":"DEBUG","message":"sending request to backend","headers":"{\"user-agent\": \"llm-ls/0.5.2; rust/unknown; ide/Neovim\", \"authorization\": \"Bearer sk-xxxx\"}","body":"{\"model\": String(\"deepseek-coder-v2:16b-lite-base-q4_0\"), \"prompt\": String(\"<|fim▁begin|>#include \\n#include \\n\\n\\n\\nclass A {\\n\\n};\\n\\nclass B {\\n\\n};\\n\\nint foo() {\\n\\n}\\n\\nint main() {\\n std::shared_ptr
a = std::make_shared();\\n std::shared_ptr b = std::make_shared();\\n <|fim▁hole|>\\n return 0;\\n}\\n<|fim▁end|>\"), \"stream\": Bool(false), \"temperature\": Number(0.2), \"top_p\": Number(0.95)}","url":"http://192.168.0.61:8080/api/v1/completions","target":"llm_ls","line_number":255,"spans":[{"request_id":"e26d9290-1619-4990-9df8-b8b665f7621d","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:03.532185Z","level":"DEBUG","message":"reuse idle connection for (\"http\", 192.168.0.61:8080)","target":"hyper::client::pool","line_number":250,"spans":[{"request_id":"e26d9290-1619-4990-9df8-b8b665f7621d","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:03.532446Z","level":"DEBUG","message":"flushed 643 bytes","target":"hyper::proto::h1::io","line_number":342}
{"timestamp":"2024-09-18T20:44:03.550477Z","level":"DEBUG","message":"parsed 5 headers","target":"hyper::proto::h1::io","line_number":207}
{"timestamp":"2024-09-18T20:44:03.550580Z","level":"DEBUG","message":"incoming body is content-length (22 bytes)","target":"hyper::proto::h1::conn","line_number":222}
{"timestamp":"2024-09-18T20:44:03.550620Z","level":"DEBUG","message":"incoming body completed","target":"hyper::proto::h1::conn","line_number":298}
{"timestamp":"2024-09-18T20:44:03.550722Z","level":"DEBUG","message":"pooling idle connection for (\"http\", 192.168.0.61:8080)","target":"hyper::client::pool","line_number":376,"spans":[{"request_id":"e26d9290-1619-4990-9df8-b8b665f7621d","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:03.550849Z","level":"ERROR","err_msg":"serde json error: data did not match any variant of untagged enum OpenAIAPIResponse","target":"llm_ls::error","line_number":8,"spans":[{"request_id":"e26d9290-1619-4990-9df8-b8b665f7621d","name":"completion_request"}]}
```

... If you've got suggestions on a better alternative to open webui, I might consider it. But I want to open the server to a few friends and don't want the hassle of manually managing api keys for raw ollama instance (that works fine btw.).

Any help would be appreciated!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the authenticated request to the Open WebUI endpoint shown in the logs and inspect how the response is deserialized into OpenAIAPIResponse. The issue is resolved when API-key-authenticated responses from the configured backend parse without the serde error and completion requests continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
ollama, rust
Domain
api, authentication, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.