apache / apache/apisix

feat(ai-proxy): add defaults field for fallback model parameters

Open
#13,149 0 comments 0 reactions 0 assignees View on GitHub
enhancement plugin
Dominant language
Lua
Stars
17.1k
Forks
2.9k
Avg merge
3d 16h
Merged PRs (30d)
63

Description

## Summary

Add a `defaults` field to the `ai-proxy` and `ai-proxy-multi` plugins that provides fallback model parameters applied only when the client request does not specify them.

## Motivation

Currently, the `options` field always overrides request values regardless of what the client sends. This is useful for enforcing parameters, but there is no way to provide sensible defaults that the client can still override.

A common use case is an LLM gateway where the operator wants to set default values (e.g., `max_tokens`, `temperature`) without restricting clients from customizing these values in their requests.

## Proposed Design

Add a `defaults` field alongside the existing `options` field:

```yaml
plugins:
ai-proxy:
model:
provider: openai
name: gpt-4o
defaults:
max_tokens: 512
temperature: 0.7
options:
model: gpt-4o # always enforced
```

**Priority order** (highest to lowest):
1. `options` — always overrides, even if the client sends a value
2. client request body — client-provided values
3. `defaults` — fallback, applied only when the client omits the field

## Difference from `options`

| Field | Behavior |
|-------|----------|
| `options` | Always overrides request values (enforcement) |
| `defaults` | Only applied when the client does not set the field (fallback) |

## Examples

Given `defaults: { max_tokens: 512, temperature: 0.7 }` and `options: { model: gpt-4o }`:

| Client sends | Effective value |
|-------------|-----------------|
| `max_tokens: 1024` | `1024` (client wins over defaults) |
| _(no max_tokens)_ | `512` (defaults applied) |
| `model: gpt-3.5-turbo` | `gpt-4o` (options always wins) |

## Implementation

A working implementation is available in PR #12895. Happy to revise based on community feedback.

Contributor guide

Open the contributing guide

Research direction

Read the existing options handling in the ai-proxy and ai-proxy-multi plugins, then compare it with the working implementation in PR #12895. Verify that defaults apply only when the client omits a parameter, client values override defaults, and options still take precedence; cover these priority cases before considering the feature complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
lua
Domain
api
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.