apache / apache/tooling-llmao

Two instances of one model do not need artificial model names

Open
#22 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

From the design discussion: *"it does appear we need to create artificial model
names, so that we can run several instances of (say) Qwen3."*

That is not required, and building on it would produce a worse catalog.
**LiteLLM load-balances across deployments sharing a `model_name`** — that is
what a model group is, and `router_utils/cooldown_handlers.py` reasons about
`len(model_group) == 1` as the single-deployment special case precisely because
more than one is the normal case.

So five Qwen3 boxes serving identical config are five routes, all
`model_name: qwen3-8b`, different `api_base`. Callers see one model; LiteLLM
picks a backend.

Artificial names are only needed when the difference is **caller-visible** — a
different context window, a different reasoning parser, anything that changes
the shape of a response. Those genuinely are different models from a caller's
point of view and deserve distinct names in the catalog.

The `[model, port, name]` third element is a different thing: it distinguishes
two vLLM *processes on one box*, which need separate ports and supervisor
units. That is box-level identity, not a LiteLLM `model_name`.

**Worth writing down**, because the two readings lead to very different
catalogs: one where `qwen3-8b` is a single offering with N backends, and one
where callers must choose between `qwen3-8b-a`, `-b` and `-c` for no reason
they can perceive.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with router_utils/cooldown_handlers.py and the model-group behavior described in the issue. Document the distinction between shared LiteLLM model names, caller-visible model differences, and the [model, port, name] process identity; done means the catalog guidance clearly explains when artificial names are and are not needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.