Two instances of one model do not need artificial model names
- Dominant language
- Python
- Stars
- 1
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
From the design discussion: *"it does appear we need to create artificial model
names, so that we can run several instances of (say) Qwen3."*
That is not required, and building on it would produce a worse catalog.
**LiteLLM load-balances across deployments sharing a `model_name`** — that is
what a model group is, and `router_utils/cooldown_handlers.py` reasons about
`len(model_group) == 1` as the single-deployment special case precisely because
more than one is the normal case.
So five Qwen3 boxes serving identical config are five routes, all
`model_name: qwen3-8b`, different `api_base`. Callers see one model; LiteLLM
picks a backend.
Artificial names are only needed when the difference is **caller-visible** — a
different context window, a different reasoning parser, anything that changes
the shape of a response. Those genuinely are different models from a caller's
point of view and deserve distinct names in the catalog.
The `[model, port, name]` third element is a different thing: it distinguishes
two vLLM *processes on one box*, which need separate ports and supervisor
units. That is box-level identity, not a LiteLLM `model_name`.
**Worth writing down**, because the two readings lead to very different
catalogs: one where `qwen3-8b` is a single offering with N backends, and one
where callers must choose between `qwen3-8b-a`, `-b` and `-c` for no reason
they can perceive.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with router_utils/cooldown_handlers.py and the model-group behavior described in the issue. Document the distinction between shared LiteLLM model names, caller-visible model differences, and the [model, port, name] process identity; done means the catalog guidance clearly explains when artificial names are and are not needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100