Allow Separate Base URLs for Strong and Weak Models in Controller
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5.5k
- Forks
- 433
- PR merge metrics
- No merged PRs in 30d
Description
For my use case, I have different models running on different servers which all replicate the OpenAI completions endpoint. However, from what I can see, it is currently not possible to use both the default OpenAI base URL and a custom server's base URL when defining the controller (or two separate server base URLs). There is functionality to create a `Controller` client from within a local server running a model, but what if there is no access to run the RouteLLM code from within the server hosting the model.
It would be great if the controller could be provided with a separate base URL for a strong and weak model as, from my understanding, right now, if a base_url is provided, it overrides the base_url used for both the strong and weak model.
**Example Use Case:**
I want to use an OpenAI model like GPT-4o (using the OpenAI completions endpoint) and an open-source model like Mistral running on a custom server with a custom URL (replicating the OpenAI completion endpoint).
**Actual Behavior:**
- Without providing a base_url parameter to the Controller class, the controller cannot call the Mistral model as it defaults to using the official OpenAI completions endpoint.
- If a custom base_url is provided, the Mistral model works but GPT-4o does not, as the GPT-4o model is not found at this new endpoint.
**Steps to Reproduce:**
- Define a controller without a base_url parameter.
- Attempt to call a model (e.g., Mistral) hosted on a custom server with its own URL.
- Define a controller with a custom base_url.
- Attempt to call an OpenAI model (e.g., GPT-4o).
**Current Workaround:**
I am currently solving this in a rudimentary way by checking whether the model called in the `Controller.completion` function (found within `kwargs["model"]`) matches the `strong_model` string or the `weak_model` string and using the corresponding base_url ive provided for each model.
**Proposed Solution:**
Introduce functionality in the Controller class to allow specifying separate base URLs for the strong and weak models.
```
controller = Controller(
strong_model="gpt-4o",
weak_model="openai/mistral",
strong_model_base_url="https://api.openai.com/v1" # or just None,
weak_model_base_url="http://custom-endpoint.com/v1"
)
```
Not sure if I am just misunderstanding something and this functionality does exist. Thank you!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Controller class and its completion function, where kwargs["model"] is used to distinguish the strong and weak models. Trace how the current base_url is applied, then add coverage showing that each model uses its own endpoint while the existing default behavior remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100