feat: support client-driven model selection in ai-proxy-multi via `models` request body field
- Dominant language
- Lua
- Stars
- 17.1k
- Forks
- 2.9k
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 63
Description
## Description
Currently, `ai-proxy-multi` selects AI instances entirely server-side via configured `priority`, `weight`, and `fallback_strategy`. The client has no way to influence which model or provider handles their request, or in what order fallbacks should be attempted.
This proposal adds support for a `models` field in the request body, allowing clients to specify their preferred model ordering while the gateway retains full control over authentication, rate limiting, and provider configuration.
## Proposed API
Clients include a `models` array in the chat completions request body. Each element can be:
**Object form** (full control):
```json
{
"messages": [{"role": "user", "content": "Hello"}],
"models": [
{"provider": "anthropic", "model": "claude-sonnet-4-20250514"},
{"provider": "openai", "model": "gpt-4o"}
]
}
```
**String shorthand** (just model names, matched against configured instances):
```json
{
"messages": [{"role": "user", "content": "Hello"}],
"models": ["claude-sonnet-4-20250514", "gpt-4o"]
}
```
**Mixed**:
```json
{
"messages": [{"role": "user", "content": "Hello"}],
"models": [
{"provider": "anthropic", "model": "claude-sonnet-4-20250514"},
"gpt-4o"
]
}
```
## Behavior
1. The `models` array defines the **client's preferred instance ordering** (first element = highest priority).
2. Each entry is matched against the route's configured `ai-proxy-multi` instances by `model` name and optionally `provider`.
3. String entries match by model name only. Object entries can match by `provider` + `model` for disambiguation (e.g., two providers serving the same model).
4. Unmatched entries are ignored (the client cannot introduce providers/models not configured on the route).
5. Configured instances not referenced in `models` are appended after the client's preferred list, in their original priority order.
6. The existing `fallback_strategy` still applies — if the top-priority instance fails, the plugin falls back through the client-specified order, then server-configured instances.
7. Auth, rate limiting, and all other instance config remain server-controlled.
## Configuration
A new optional boolean on the plugin config to opt-in:
```yaml
plugins:
ai-proxy-multi:
allow_client_model_preference: true # default: false
instances:
- name: anthropic-primary
provider: anthropic
options:
model: claude-sonnet-4-20250514
# ...
- name: openai-fallback
provider: openai
options:
model: gpt-4o
# ...
```
When `allow_client_model_preference` is `false` (default), the `models` field is stripped from the request body and instance ordering is purely server-driven. This ensures backward compatibility.
## Use Case
We're building an LLM proxy gateway where multiple teams consume AI services through a shared gateway. Different clients have different model preferences:
- Team A prefers Claude with GPT-4o fallback
- Team B prefers GPT-4o with Claude fallback
- The gateway team manages auth keys, rate limits, and provider configs centrally
Today this requires separate routes per team/preference. With client-driven model selection, a single route handles all teams while respecting their preferences.
## Implementation Notes
- The matching logic would live in `ai-proxy-multi.lua`, executed before instance selection in the `access` phase.
- The `models` field should be stripped from the request body before forwarding to the upstream provider (providers don't recognize it).
- String matching should be case-sensitive to match provider model naming conventions.
- When using `chash` balancer, client preference should take precedence over hash-based selection.
I'm happy to submit a PR for this feature if the approach looks good.
Contributor guide
Research direction
Start by reading ai-proxy-multi.lua and its access-phase instance-selection flow. Trace how request bodies and configured instances are handled before upstream forwarding, then define tests for opt-in ordering, string and object matching, unmatched entries, stripping models, and fallback behavior; done means the default remains server-driven while enabled routes honor client preferences.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- lua
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100