RFC: LLM inference gateway with structured trace stream for Endpoints/Deployments

## Summary

Backend.AI exposes inference workloads through `Endpoint` / `Deployment`, but there is no standardized layer in front of those endpoints to:

- emit structured per-call trace events (model, prompt/completion size, latency, token counts, finish reason, cost),
- route requests across multiple endpoints (and optionally external providers) by policy (cost / latency / quality / SLO),
- enforce pre/post-call validators (PII redaction, schema, jailbreak detection, etc.),
- meter token + accelerator-hour usage per caller / project.

This RFC proposes an **LLM Inference Gateway** as a first-class Backend.AI component sitting between callers and inference targets, with a queryable + streaming trace API for downstream consumers.

## Motivation

Inference traffic is increasingly the dominant workload on Backend.AI clusters. Today, operators have:

- only generic resource metrics (CPU/GPU/memory) — no model-aware metrics
- no canonical place to apply routing, fallback, retries, or cost-aware switching
- no shared format for per-call observability — each user reinvents tracing inside their own client code
- no native budget enforcement at the inference boundary

A single gateway resolves all four concerns and gives every Backend.AI deployment uniform observability and policy without each caller having to instrument.

## Non-goals

- Building a serving runtime — defer to existing `Endpoint` / `Deployment` / `RuntimeVariant`.
- Building a model registry — defer to `ModelCard` / `ArtifactRegistry`.
- Replacing the existing scheduler — gateway is purely on the request data plane.
- Building offline evaluation — that is a workflow concern; the gateway only emits traces that an evaluator can consume.

## Proposed capabilities

### 1. Inference proxy

A managed proxy in front of inference endpoints, OpenAI-compatible at the API surface (chat/completions, embeddings, optionally responses-API style). Internally, it dispatches to:

- one or more Backend.AI `Endpoint`s,
- optionally external providers (OpenAI, Anthropic, etc.) with credentials managed via existing keypair / secret primitives.

Default behavior is transparent passthrough; routing/policy/validators are opt-in.

### 2. Structured trace event schema

For every call, emit a trace event including (at minimum):

- request id, parent span id (for nested tool calls)
- caller identity (keypair / project / domain)
- target route resolved (which endpoint / which provider / which model id)
- prompt and completion token counts, finish reason
- latency (queue, ttft, total)
- request metadata (model, temperature, max tokens, tool calls)
- cost estimate (per resolved route)
- guardrail outcomes (allowed / blocked / redacted, with rule id)
- error class and code if applicable

Bodies (prompt/completion text) should be **opt-in per project**, with redaction hooks.

### 3. Trace delivery API

Two delivery modes for downstream consumers:

- **Streaming**: durable event channel (existing event-dispatcher patterns) for near-real-time consumption.
- **Query**: REST/GraphQL surface for historical lookup by request id, caller, project, time range, model.

Retention configurable per project; raw bodies (when enabled) on a shorter TTL than metadata.

### 4. Routing policies

Declarative routing rules at the project or endpoint-set level:

- weighted split / canary / shadow
- failover order with health/error-rate thresholds
- cost ceiling / latency SLO based selection
- variant routing (e.g., A/B by request attribute)

### 5. Validators (guardrails)

Pluggable pre-request and post-response validators with a common interface:

- input: request or response payload, caller context
- output: allow / block / mutate (with reason and rule id, surfaced in trace)

Initial built-ins: PII redaction, schema validation, basic content filters. Third-party validators registered via the existing plugin system.

### 6. Metering

Per-caller, per-project, per-endpoint counters (tokens, requests, accelerator-seconds, estimated cost) exposed through the existing metrics surface, plus quota-style soft/hard limits configurable per project.

## API surface (sketch)

- `POST /v1/llm/chat/completions` — OpenAI-compatible
- `POST /v1/llm/embeddings`
- `GET /v1/llm/traces?project=...&since=...` — query
- `WS /v1/llm/traces/stream` — streaming
- GraphQL: `llmTrace(id)`, `llmTraces(filter)` connection
- GraphQL: `llmRoutingPolicy`, `llmValidator` CRUD

## Open design questions

1. Do we proxy the raw HTTP path, or also expose a native Backend.AI API form?
2. Where does the gateway run — co-located with `app-proxy`, alongside `manager`, or as a separate component?
3. Trace persistence: piggyback on the existing event store, or a dedicated time-series sink (ClickHouse / Loki / Postgres partitioned)? Volume implications.
4. Cost model: do we ship a default per-token price table per known model, allow operator override per endpoint, or both?
5. Body retention defaults: off, on, or sample-N%?
6. External-provider credential storage: extend keypair, or a new credential entity?
7. Rate limiting / fairness: gateway-local or coordinated through `manager`?
8. Backwards compatibility: do existing `Endpoint` direct callers continue to bypass the gateway, or is the gateway eventually mandatory?

## Phasing suggestion

1. **MVP** — passthrough proxy with trace emission (no routing, no validators); query + stream API.
2. **Routing** — weighted/canary/failover policies.
3. **Validators** — pluggable pre/post hooks with built-ins.
4. **External providers** — OpenAI/Anthropic/etc. behind the same surface.
5. **Quotas** — soft/hard budget enforcement.

## Why this belongs in Backend.AI

- The gateway must see *every* inference call regardless of caller — not only those originating from a specific tool. That requires sitting at the data plane.
- Routing and metering are most naturally expressed next to the existing `Endpoint`/`Deployment` and resource-policy primitives.
- Putting it anywhere else means an extra hop, partial observability, or per-tool reinvention.

Happy to iterate on the design — particularly on the open questions above.