huggingface / huggingface/llm-ls
ollama backend does not support api keys
- Dominant language
- Rust
- Stars
- 881
- Forks
- 71
- Avg merge
- 33m
- Merged PRs (30d)
- 1
Description
Hi!
I'm trying to set up llm-ls via llm.nvim plugin and I'm hitting weird serdes errors. I set the following config:
```
{
backend="openai",
url = "http://192.168.0.61:8080/api",
api_token = "sk-xxxx", -- replace after publishing outside local network
lsp = {
cmd_env = { LLM_LOG_LEVEL = "DEBUG" }
},
}
```
Weird thing is, completion seems to work just fine. But after each completion, I get a serdes error per following log. Any idea what am I doing wrong? I find it hard to find docs on using open webui as a backend, as well as the open webui docs for their API (I believe it is supposed to be openai compatible, hence I set it so).
Relevant logs (api key stripped):
```
{"timestamp":"2024-09-18T20:44:02.195454Z","level":"INFO","message":"file:///home/krakonos/test.cpp changed","target":"llm_ls","line_number":682}
{"timestamp":"2024-09-18T20:44:02.195645Z","level":"DEBUG","message":"client asked to cancel request 4, but no such pending request exists, ignoring","target":"tower_lsp::service::pending","line_number":64}
{"timestamp":"2024-09-18T20:44:02.346897Z","level":"INFO","message":"received completion request","document_url":"file:///home/krakonos/test.cpp","cursor_line":"19","cursor_character":"49","language_id":"cpp","model":"deepseek-coder-v2:16b-lite-base-q4_0","backend":"OpenAi { url: \"http://192.168.0.61:8080/api\" }","ide":"neovim","request_body":"{\"temperature\":0.2,\"top_p\":0.95}","disable_url_path_completion":false,"target":"llm_ls","line_number":514,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.347146Z","level":"INFO","message":"completion type: SingleLine","completion_type":"single_line","target":"llm_ls","line_number":537,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.347201Z","level":"INFO","message":"built prompt in 0 ms","prompt":"<|fim▁begin|>#include \n#include \n\n\n\nclass A {\n\n};\n\nclass B {\n\n};\n\nint foo() {\n\n}\n\nint main() {\n std::shared_ptr a = std::make_shared();\n std::shared_ptr b = std::make_shared();<|fim▁hole|>\n return 0;\n}\n<|fim▁end|>","build_prompt_ms":"0","target":"llm_ls","line_number":204,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.347232Z","level":"INFO","message":"sending request to backend","headers":"{\"user-agent\": \"llm-ls/0.5.2; rust/unknown; ide/Neovim\", \"authorization\": \"Bearer sk-xxxx\"}","url":"http://192.168.0.61:8080/api/v1/completions","target":"llm_ls","line_number":254,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.347281Z","level":"DEBUG","message":"sending request to backend","headers":"{\"user-agent\": \"llm-ls/0.5.2; rust/unknown; ide/Neovim\", \"authorization\": \"Bearer sk-xxxx\"}","body":"{\"model\": String(\"deepseek-coder-v2:16b-lite-base-q4_0\"), \"prompt\": String(\"<|fim▁begin|>#include \\n#include \\n\\n\\n\\nclass A {\\n\\n};\\n\\nclass B {\\n\\n};\\n\\nint foo() {\\n\\n}\\n\\nint main() {\\n std::shared_ptr a = std::make_shared();\\n std::shared_ptr b = std::make_shared();<|fim▁hole|>\\n return 0;\\n}\\n<|fim▁end|>\"), \"stream\": Bool(false), \"temperature\": Number(0.2), \"top_p\": Number(0.95)}","url":"http://192.168.0.61:8080/api/v1/completions","target":"llm_ls","line_number":255,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.347408Z","level":"DEBUG","message":"starting new connection: http://192.168.0.61:8080/","log.target":"reqwest::connect","log.module_path":"reqwest::connect","log.file":"/home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.20/src/connect.rs","log.line":429,"target":"reqwest::connect","line_number":429,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.347446Z","level":"DEBUG","message":"connecting to 192.168.0.61:8080","target":"hyper::client::connect::http","line_number":537,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.363434Z","level":"DEBUG","message":"connected to 192.168.0.61:8080","target":"hyper::client::connect::http","line_number":540,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.363816Z","level":"DEBUG","message":"flushed 637 bytes","target":"hyper::proto::h1::io","line_number":342}
{"timestamp":"2024-09-18T20:44:02.368285Z","level":"DEBUG","message":"parsed 5 headers","target":"hyper::proto::h1::io","line_number":207}
{"timestamp":"2024-09-18T20:44:02.368420Z","level":"DEBUG","message":"incoming body is content-length (22 bytes)","target":"hyper::proto::h1::conn","line_number":222}
{"timestamp":"2024-09-18T20:44:02.368492Z","level":"DEBUG","message":"incoming body completed","target":"hyper::proto::h1::conn","line_number":298}
{"timestamp":"2024-09-18T20:44:02.368619Z","level":"DEBUG","message":"pooling idle connection for (\"http\", 192.168.0.61:8080)","target":"hyper::client::pool","line_number":376,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:02.368860Z","level":"ERROR","err_msg":"serde json error: data did not match any variant of untagged enum OpenAIAPIResponse","target":"llm_ls::error","line_number":8,"spans":[{"request_id":"3d2389a7-a53a-4e09-a658-4095254c72d4","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:03.380607Z","level":"INFO","message":"file:///home/krakonos/test.cpp changed","target":"llm_ls","line_number":682}
{"timestamp":"2024-09-18T20:44:03.381212Z","level":"DEBUG","message":"client asked to cancel request 5, but no such pending request exists, ignoring","target":"tower_lsp::service::pending","line_number":64}
{"timestamp":"2024-09-18T20:44:03.531812Z","level":"INFO","message":"received completion request","document_url":"file:///home/krakonos/test.cpp","cursor_line":"20","cursor_character":"4","language_id":"cpp","model":"deepseek-coder-v2:16b-lite-base-q4_0","backend":"OpenAi { url: \"http://192.168.0.61:8080/api\" }","ide":"neovim","request_body":"{\"temperature\":0.2,\"top_p\":0.95}","disable_url_path_completion":false,"target":"llm_ls","line_number":514,"spans":[{"request_id":"e26d9290-1619-4990-9df8-b8b665f7621d","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:03.531954Z","level":"INFO","message":"completion type: SingleLine","completion_type":"single_line","target":"llm_ls","line_number":537,"spans":[{"request_id":"e26d9290-1619-4990-9df8-b8b665f7621d","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:03.532024Z","level":"INFO","message":"built prompt in 0 ms","prompt":"<|fim▁begin|>#include \n#include \n\n\n\nclass A {\n\n};\n\nclass B {\n\n};\n\nint foo() {\n\n}\n\nint main() {\n std::shared_ptr a = std::make_shared();\n std::shared_ptr b = std::make_shared();\n <|fim▁hole|>\n return 0;\n}\n<|fim▁end|>","build_prompt_ms":"0","target":"llm_ls","line_number":204,"spans":[{"request_id":"e26d9290-1619-4990-9df8-b8b665f7621d","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:03.532053Z","level":"INFO","message":"sending request to backend","headers":"{\"user-agent\": \"llm-ls/0.5.2; rust/unknown; ide/Neovim\", \"authorization\": \"Bearer sk-xxxx\"}","url":"http://192.168.0.61:8080/api/v1/completions","target":"llm_ls","line_number":254,"spans":[{"request_id":"e26d9290-1619-4990-9df8-b8b665f7621d","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:03.532073Z","level":"DEBUG","message":"sending request to backend","headers":"{\"user-agent\": \"llm-ls/0.5.2; rust/unknown; ide/Neovim\", \"authorization\": \"Bearer sk-xxxx\"}","body":"{\"model\": String(\"deepseek-coder-v2:16b-lite-base-q4_0\"), \"prompt\": String(\"<|fim▁begin|>#include \\n#include \\n\\n\\n\\nclass A {\\n\\n};\\n\\nclass B {\\n\\n};\\n\\nint foo() {\\n\\n}\\n\\nint main() {\\n std::shared_ptr a = std::make_shared();\\n std::shared_ptr b = std::make_shared();\\n <|fim▁hole|>\\n return 0;\\n}\\n<|fim▁end|>\"), \"stream\": Bool(false), \"temperature\": Number(0.2), \"top_p\": Number(0.95)}","url":"http://192.168.0.61:8080/api/v1/completions","target":"llm_ls","line_number":255,"spans":[{"request_id":"e26d9290-1619-4990-9df8-b8b665f7621d","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:03.532185Z","level":"DEBUG","message":"reuse idle connection for (\"http\", 192.168.0.61:8080)","target":"hyper::client::pool","line_number":250,"spans":[{"request_id":"e26d9290-1619-4990-9df8-b8b665f7621d","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:03.532446Z","level":"DEBUG","message":"flushed 643 bytes","target":"hyper::proto::h1::io","line_number":342}
{"timestamp":"2024-09-18T20:44:03.550477Z","level":"DEBUG","message":"parsed 5 headers","target":"hyper::proto::h1::io","line_number":207}
{"timestamp":"2024-09-18T20:44:03.550580Z","level":"DEBUG","message":"incoming body is content-length (22 bytes)","target":"hyper::proto::h1::conn","line_number":222}
{"timestamp":"2024-09-18T20:44:03.550620Z","level":"DEBUG","message":"incoming body completed","target":"hyper::proto::h1::conn","line_number":298}
{"timestamp":"2024-09-18T20:44:03.550722Z","level":"DEBUG","message":"pooling idle connection for (\"http\", 192.168.0.61:8080)","target":"hyper::client::pool","line_number":376,"spans":[{"request_id":"e26d9290-1619-4990-9df8-b8b665f7621d","name":"completion_request"}]}
{"timestamp":"2024-09-18T20:44:03.550849Z","level":"ERROR","err_msg":"serde json error: data did not match any variant of untagged enum OpenAIAPIResponse","target":"llm_ls::error","line_number":8,"spans":[{"request_id":"e26d9290-1619-4990-9df8-b8b665f7621d","name":"completion_request"}]}
```
... If you've got suggestions on a better alternative to open webui, I might consider it. But I want to open the server to a few friends and don't want the hassle of manually managing api keys for raw ollama instance (that works fine btw.).
Any help would be appreciated!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the authenticated request to the Open WebUI endpoint shown in the logs and inspect how the response is deserialized into OpenAIAPIResponse. The issue is resolved when API-key-authenticated responses from the configured backend parse without the serde error and completion requests continue to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ollama, rust
- Domain
- api, authentication, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100