apache / apache/apisix

feat: support client-driven model selection in ai-proxy-multi via `models` request body field

Open
#13,083 3 comments 0 reactions 0 assignees View on GitHub
feature-request
Dominant language
Lua
Stars
17.1k
Forks
2.9k
Avg merge
3d 16h
Merged PRs (30d)
63

Description

## Description

Currently, `ai-proxy-multi` selects AI instances entirely server-side via configured `priority`, `weight`, and `fallback_strategy`. The client has no way to influence which model or provider handles their request, or in what order fallbacks should be attempted.

This proposal adds support for a `models` field in the request body, allowing clients to specify their preferred model ordering while the gateway retains full control over authentication, rate limiting, and provider configuration.

## Proposed API

Clients include a `models` array in the chat completions request body. Each element can be:

**Object form** (full control):
```json
{
"messages": [{"role": "user", "content": "Hello"}],
"models": [
{"provider": "anthropic", "model": "claude-sonnet-4-20250514"},
{"provider": "openai", "model": "gpt-4o"}
]
}
```

**String shorthand** (just model names, matched against configured instances):
```json
{
"messages": [{"role": "user", "content": "Hello"}],
"models": ["claude-sonnet-4-20250514", "gpt-4o"]
}
```

**Mixed**:
```json
{
"messages": [{"role": "user", "content": "Hello"}],
"models": [
{"provider": "anthropic", "model": "claude-sonnet-4-20250514"},
"gpt-4o"
]
}
```

## Behavior

1. The `models` array defines the **client's preferred instance ordering** (first element = highest priority).
2. Each entry is matched against the route's configured `ai-proxy-multi` instances by `model` name and optionally `provider`.
3. String entries match by model name only. Object entries can match by `provider` + `model` for disambiguation (e.g., two providers serving the same model).
4. Unmatched entries are ignored (the client cannot introduce providers/models not configured on the route).
5. Configured instances not referenced in `models` are appended after the client's preferred list, in their original priority order.
6. The existing `fallback_strategy` still applies — if the top-priority instance fails, the plugin falls back through the client-specified order, then server-configured instances.
7. Auth, rate limiting, and all other instance config remain server-controlled.

## Configuration

A new optional boolean on the plugin config to opt-in:

```yaml
plugins:
ai-proxy-multi:
allow_client_model_preference: true # default: false
instances:
- name: anthropic-primary
provider: anthropic
options:
model: claude-sonnet-4-20250514
# ...
- name: openai-fallback
provider: openai
options:
model: gpt-4o
# ...
```

When `allow_client_model_preference` is `false` (default), the `models` field is stripped from the request body and instance ordering is purely server-driven. This ensures backward compatibility.

## Use Case

We're building an LLM proxy gateway where multiple teams consume AI services through a shared gateway. Different clients have different model preferences:
- Team A prefers Claude with GPT-4o fallback
- Team B prefers GPT-4o with Claude fallback
- The gateway team manages auth keys, rate limits, and provider configs centrally

Today this requires separate routes per team/preference. With client-driven model selection, a single route handles all teams while respecting their preferences.

## Implementation Notes

- The matching logic would live in `ai-proxy-multi.lua`, executed before instance selection in the `access` phase.
- The `models` field should be stripped from the request body before forwarding to the upstream provider (providers don't recognize it).
- String matching should be case-sensitive to match provider model naming conventions.
- When using `chash` balancer, client preference should take precedence over hash-based selection.

I'm happy to submit a PR for this feature if the approach looks good.

Contributor guide

Open the contributing guide

Research direction

Start by reading ai-proxy-multi.lua and its access-phase instance-selection flow. Trace how request bodies and configured instances are handled before upstream forwarding, then define tests for opt-in ordering, string and object matching, unmatched entries, stripping models, and fallback behavior; done means the default remains server-driven while enabled routes honor client preferences.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua
Domain
api, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.