pathwaycom / pathwaycom/serviette

Bug: `type: litellm` LLM never routed through LiteLLM

Open
#2 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
5
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Note : Below is a AI generated human reviewed bug report for a human discovered bug, I encountered the bug and had to fix it, I asked IA to document it.

If the quality is bad feel free to say so (I'll learn), or reject.

I use the fix, it is proven to work, the code changes are sound and reviewed. We can discuss about the asserts in the code (and remove them) but the bug is real and the fix make the code work according to the documentation.

I open the pull request right after pushing the issue.

Symptom

Using the webui LLM dialog (/api/v1/rag) with a litellm-typed config:

llm:
  type: litellm
  model: "openrouter/moonshotai/kimi-k3"
  api_key: "sk-or-..."

the request went to api.openai.com with the OpenRouter key and failed:

openai.AuthenticationError: Error code: 401 - Incorrect API key provided ...
  at serviette/server/main.py:243  -> llm.complete(...)
  at serviette/server/llm.py:122   -> client.chat.completions.create(...)

LiteLLM was never initialized on this path.

Root cause

build_llm in packages/serviette/serviette/server/llm.py treated litellm
as OpenAI-compatible:

_OPENAI_COMPATIBLE = {"openai", "litellm"}

def build_llm(config):
    ...
    if config.type in _OPENAI_COMPATIBLE:
        return OpenAIChat(config)

OpenAIChat._ensure_client builds AsyncOpenAI(api_key=..., **extra). With
this config extra is empty, so base_url defaults to
https://api.openai.com. The provider prefix in model
(openrouter/...) was passed verbatim to OpenAI, which knows neither the
model nor the key → 401.

The config itself is valid — the quickstart wizard
(quickstart/wizard.py:607) offers litellm as an LLM type, so this was a
server-side misrouting, not a config error.

A second, latent instance of the same bug existed in
server/reranker.py: LLMReranker._ensure_chat constructed OpenAIChat
directly, ignoring a litellm type inherited from the top-level llm
section.

Fix

  • server/llm.py — added a LiteLLMChat backend that calls
    litellm.acompletion(model=..., api_key=..., messages=...), so the
    provider prefix in model selects the endpoint (OpenRouter here) and the
    key is forwarded to that provider. build_llm now dispatches:
    mockMockLLM, litellmLiteLLMChat, openaiOpenAIChat.
    The openai path is unchanged.
  • server/reranker.py_ensure_chat now uses build_llm(...)
    instead of hardcoding OpenAIChat, so an LLM reranker honors the
    configured type.
  • tests/test_server.py — regression test
    test_litellm_routes_through_litellm_not_openai_client: drives
    /api/v1/rag with a litellm LLM, patches litellm.acompletion, and
    asserts the outgoing call carries the provider-prefixed model and the
    API key, with no base_url (i.e. no OpenAI client involved).

Verification

  • 43 tests pass (test_server.py, test_rag_quality.py,
    test_frontend.py, test_qdrant_hybrid.py), including the new
    regression test.
  • Manual check: build_llm(LLMConfig(type="litellm", ...)) returns
    LiteLLMChat; mocked litellm.acompletion receives
    model="openrouter/moonshotai/kimi-k3" and the configured api_key.

Note: embedder.py intentionally keeps its own
_OPENAI_COMPATIBLE = {"openai", "litellm"} — server-side embedders
genuinely map litellm onto the OpenAI-compatible async client; that path
is unrelated and untouched.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/serviette/serviette/server/llm.py and the existing build_llm dispatch, then inspect server/reranker.py for its chat construction. Run tests/test_server.py, especially test_litellm_routes_through_litellm_not_openai_client. Done means the litellm configuration uses the intended provider path in both server flows while the existing OpenAI path remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.