apache / apache/tooling-llmao

Routes should use the hosted_vllm/ provider prefix, not openai/

Open
#34 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Every self-hosted route is created with `litellm_params.model` set to
`openai/`:

```json
"key": "openai/qwen3.8-27b",
"litellm_provider": "openai"
```

LiteLLM's vLLM docs say the correct prefix for an OpenAI-compatible vLLM
server is **`hosted_vllm/`**. `openai/` tells LiteLLM it is talking to OpenAI
itself.

> Provider Route on LiteLLM: `hosted_vllm/` (for OpenAI compatible server),
> `vllm/` ([DEPRECATED] for vLLM sdk usage)

## Why it matters — two problems it likely explains

**`reasoning_effort` is validated against the wrong vocabulary.** Claude Code
sends `reasoning_effort: high`, which OpenAI accepts and Qwen3.8-27B does
not:

```
400 OpenAIException - {"error":{"message":"Unexpected reasoning effort high.
Supported types are xhigh (default), medium, and low."}}
```

LiteLLM passed it straight through because its capability map for the
`openai` provider says the parameter is supported. `drop_params: true` is
already on and did not help, for the same reason — there was nothing LiteLLM
believed needed dropping.

The docs show `reasoning_effort` as a first-class `hosted_vllm/` feature, so
the provider has its own handling for it.

**And backend errors may translate badly.** We have seen a vLLM 422 arrive as
HTTP 200 with a body of literal `null`, and a 400 arrive as
`litellm.InternalServerError` with a truncated message. A provider that knows
it is speaking to vLLM should map these more faithfully than one assuming
OpenAI's error shapes.

Worth testing rather than assuming, but both symptoms are consistent with the
wrong provider.

## Where to change it

**`llmao/litellm_client.py`** — wherever the `/model/new` payload is built.
`litellm_params.model` currently gets `openai/` prefixed to the served model
name; it should be `hosted_vllm/`.

This is route-creation code rather than config, so existing routes are
unaffected until they are recreated. Worth deleting and re-pushing after the
change, or the fleet ends up with a mix.

**`model_list.yaml.example`** — each entry has
`litellm_params.model: openai/`. That value is copied into the route at
registration, so it needs the same change.

**`litellm.yaml.example`** if any sample route carries the prefix.

**`docs/manual-fleet-notes.md`** should record why, since `openai/` is the
intuitive-looking choice and someone will otherwise change it back.

## How to verify before committing

Create a parallel route and compare, without touching the live one:

```bash
curl -sS -X POST http://127.0.0.1:4000/model/new \
-H "Authorization: Bearer $MK" -H 'Content-Type: application/json' \
-d '{"model_name":"qwen-hv-test","litellm_params":{"model":"hosted_vllm/qwen3.8-27b","api_base":"http://103.196.86.105:15602/v1","api_key":"'"$VLLM_API_KEY"'"}}'

curl -sS http://127.0.0.1:4000/v1/chat/completions \
-H "Authorization: Bearer $MK" -H 'Content-Type: application/json' \
-d '{"model":"qwen-hv-test","messages":[{"role":"user","content":"Say OK"}],"reasoning_effort":"high","max_tokens":20}'
```

If that succeeds where the `openai/` route 400s, the prefix is the fix.

Also worth re-testing through it: tool calling, the Anthropic `/v1/messages`
translation, vision with a base64 data URL, and an error case. Remove the
test route afterwards with `/model/delete`.

## Note

The docs also list `/rerank`, `/embeddings` and `/audio/transcriptions` as
supported on `hosted_vllm/`, plus video input via the unified files message.
None are in scope now, but they are only reachable under the correct prefix —
so this closes off capabilities as well as causing the bugs above.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in llmao/litellm_client.py at the /model/new payload construction, then inspect the matching entries in model_list.yaml.example and litellm.yaml.example. Create a parallel test route with the hosted_vllm/ payload and run the supplied chat completion curl, including reasoning_effort and an error case. Update the documented examples and docs/manual-fleet-notes.md, and verify the affected route behavior before removing the test route.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.