Dstack-TEE / Dstack-TEE/private-ai-gateway
Declare upstream API surfaces for native passthrough vs conversion
- Dominant language
- Rust
- Stars
- 32
- Forks
- 8
- Avg merge
- 11h 9m
- Merged PRs (30d)
- 36
Description
## Summary
The gateway should choose native forwarding or conversion from explicit API capabilities on each model route. The current `format: "openai" | "anthropic"` field loses an important distinction: an OpenAI-compatible upstream may implement Chat Completions, Responses, or both.
A provider name describes the vendor or compatibility family. An API surface describes the wire contract. Routing needs the latter.
## Current problem
The incoming path tells us the downstream API surface, but the control plane only returns a coarse provider format. For example, `/v1/responses` plus `format: "openai"` causes the gateway to prepare a Responses request and send it to `/v1/responses`. This assumes the selected upstream implements that endpoint.
This becomes a problem when:
- A vLLM, SGLang, or other OpenAI-compatible route implements `/v1/chat/completions` but not `/v1/responses`. The gateway contacts an endpoint that is known neither to exist nor to support the request.
- An upstream supports Responses natively, but the gateway selects a conversion path. Tool call IDs, item IDs, reasoning items, encrypted reasoning content, or SSE event types may be lost or rewritten unnecessarily.
- The endpoint exists but lacks a requested feature, such as reasoning, parallel tool calls, or streaming tool arguments. Endpoint support alone does not guarantee semantic support.
- Independently configured `provider`, `format`, and `path` values disagree. Request shaping, authentication, upstream path selection, and response parsing can then follow different assumptions.
Runtime probing or retrying another format after a 404 would make behavior request-dependent and could submit the same request twice. Support should be declared before the request is sent.
## Proposed configuration model
Use `API surface` consistently for a request and response wire contract:
```ts
type ApiSurface =
| "openai-chat-completions"
| "openai-responses"
| "anthropic-messages";
type ApiFeature =
| "streaming"
| "tools"
| "parallel-tools"
| "reasoning"
| "encrypted-reasoning";
interface ApiBinding {
surface: ApiSurface;
path: string;
features: ApiFeature[];
requestPolicy: string;
}
interface ModelRoute {
routeId: string;
upstreamModel: string;
apiBindings: ApiBinding[];
}
interface RouteCandidate {
routeId: string;
upstreamSurface: ApiSurface;
engine?: "sglang" | "vllm";
}
```
Example:
```json
{
"routeId": "openai:gpt-5",
"upstreamModel": "gpt-5",
"apiBindings": [
{
"surface": "openai-chat-completions",
"path": "/v1/chat/completions",
"features": ["streaming", "tools", "parallel-tools"],
"requestPolicy": "openai-chat-default"
},
{
"surface": "openai-responses",
"path": "/v1/responses",
"features": ["streaming", "tools", "parallel-tools", "reasoning", "encrypted-reasoning"],
"requestPolicy": "openai-responses-default"
}
]
}
```
Bindings should live in one authoritative route configuration. `provider` may remain for authentication and verification, but it should not imply an API surface. Paths should belong to bindings rather than exist as a separate claim about the route.
The feature list should cover semantic capabilities needed for safe routing and conversion. It should not duplicate every request parameter.
## Request planning
Build one request plan before contacting the upstream:
```ts
interface RequestPlan {
downstreamSurface: ApiSurface;
upstreamSurface: ApiSurface;
upstreamPath: string;
mode:
| { kind: "native" }
| { kind: "convert"; converter: string };
requestPolicy: string;
}
```
Planning rules:
1. Derive the downstream surface from the incoming endpoint.
2. Select an upstream binding that supports the required features.
3. If the surfaces match, use native mode.
4. Otherwise, use conversion mode only when a registered converter can represent every requested construct on the target surface.
5. Reject the request before any upstream call when neither option is valid.
The resulting plan must drive request shaping, upstream path selection, response parsing, SSE handling, and error translation. Those decisions should not be recomputed independently later in the pipeline.
## Whitelisting
Native mode does not mean raw forwarding. The gateway must still apply the selected binding's outbound request policy so security-sensitive flags cannot reach the upstream.
A policy should explicitly define its allowed fields and what happens to other fields:
```ts
interface RequestPolicy {
allowedFields: string[];
unknownFieldAction: "drop" | "reject";
}
```
For allowed fields, native mode should preserve values exactly, especially nested tool definitions, tool call identifiers, reasoning items, and opaque reasoning content. Conversion mode should validate representability, convert once, then apply the target binding's outbound policy. Silent semantic loss is not an acceptable conversion.
## Acceptance criteria
- A Responses request cannot reach a Chat-Completions-only binding.
- Native Responses and Messages preserve allowed tool call and reasoning content without schema rewriting.
- Disallowed fields are dropped or rejected exactly as declared by the selected request policy.
- Missing endpoint or feature support is rejected before an upstream request is made.
- Conversion happens only through an explicit converter, and unrepresentable content produces a clear client error.
- Configuration validation catches duplicate or contradictory bindings.
- `api-fidelity-tests/` covers native forwarding, policy filtering, supported conversion, unsupported conversion, tool calls, reasoning content, and streaming events.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the existing route configuration and request-planning pipeline described in the issue, then inspect api-fidelity-tests/. Map the current provider, format, and path decisions to the proposed bindings and plan. Done means validation rejects unsupported or contradictory routes before an upstream call, while the listed native, policy, conversion, tool, reasoning, and streaming cases are covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, typescript
- Domain
- api, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100