Endpoint composition primitives: sidecars, internal circuits, and upstream binding

## Summary

Today a [http://Backend.AI](http://Backend.AI) model service is exactly one container behind exactly one publicly reachable circuit. Any "helper" process that needs to run alongside the inference runtime — LLM guardrail, PII filter, semantic cache, A/B router, LoRA adapter multiplexer, token-counting / billing proxy, traffic mirror — forces operators into one of three unsatisfying choices:

1. Fuse helper and model into one image (couples lifecycles, wastes GPU across replicas).
1. Deploy two endpoints and trust a JWT on a public URL (token-gated ≠ network-isolated).
1. Embed logic in the frontend / client (breaks server-side guarantees).

This Epic proposes three **generic** primitives that together unlock all of the above without the platform learning about any specific use case (no `safety_policy`, no `is_guardrail`, no protocol-aware proxy middleware):

1. **Sidecar kernels in a model endpoint** — heterogeneous helpers co-located with `main` on the existing per-session bridge network.
1. **Internal-only circuits in AppProxy** — endpoints reachable only from a cluster-local URL, never from the public frontend.
1. `upstream_endpoint_id` on `EndpointRow` — a declarative pointer that lets the manager broker tokens, couple lifecycles, and surface lineage between two endpoints.

Each phase is independently shippable and independently useful. Motivating use case is LLM guardrails, but the schema stays domain-neutral.

## Motivation

### Use cases that all collapse onto the same primitives

|Use case|Shape|
|---|---|
|LLM safety rails (LlamaGuard, NeMo) co-hosted with a small model|Sidecar next to main|
|Shared, GPU-heavy guardrail fronting many models|Internal upstream + per-model gateway|
|Semantic / KV cache|Sidecar next to main|
|A/B router for model revisions|Gateway endpoint with two upstreams|
|Token-count / billing middleware|Gateway sidecar|
|Traffic mirroring to a shadow model|Sidecar + internal upstream|
|LoRA adapter selector|Gateway sidecar with main as base model|

### What current primitives can't express

- `EndpointRow` is single-kernel by construction at the modeling layer. `cluster_size` is accepted in `NewServiceRequestModel` and persisted onto the revision, but no code path routes traffic to anything other than `main`, and `model-definition.yaml` cannot declare a second process with a different image/command.
- `open_to_public=false` means **auth-gated**, not **network-isolated**: the `Circuit` is still opened on an external `Worker` frontend listener and the URL is publicly resolvable. Grep across `src/ai/backend/appproxy/` finds no existing `internal` / `private` / `loopback` circuit concept.
- There is no first-class relation between two endpoints. Anything gateway-like has to be coordinated by an external orchestrator minting tokens, injecting upstream URLs, and tracking the pair's lifecycle by hand.

## Goals

- Compose multiple containers inside one model endpoint where they must share a fate and a local network.
- Make "this endpoint cannot be reached from the public internet" expressible in the schema, not a convention.
- Express "endpoint A is fronted by endpoint B" declaratively so the manager handles token provisioning, cascade delete, and lineage.
- Keep AppProxy transport-level. No protocol parsing, no payload inspection.

## Non-goals

- **No guardrail/safety domain types.** No `safety_policy`, `content_filter`, `is_guardrail` columns or enums — ever.
- **No pipeline DAG primitive in the manager.** Multi-stage composition is achieved by chaining gateways.
- **No protocol-aware AppProxy middleware.** Application logic (token counting, message inspection) lives in a sidecar container.
- **No blessed safety model images** shipped by the platform.
- **No** `MULTI_NODE` + sidecars in phase 1. Sidecars are co-located with `main`; multi-node is a later extension.

## Phased rollout

Each phase will be filed as a child Task linked to this Epic.

### Phase 1 — Sidecar kernels in a model endpoint

Extends `ExecutionSpec` with an optional `sidecars: list[SidecarSpec]`. Reuses the existing single-node bridge network (`bai-singlenode-{session_id`}) that already places kernels with stable hostnames via Docker endpoint aliases. Introduces:

- `cluster_role="sidecar"` (alongside `main` / `sub`) with hostname `sidecar-{name`}.
- `is_endpoint_frontend: bool` on kernel creation config — the scheduler picks the first sidecar with `role="gateway"` if any exists, else `main` (backward-compatible default). `RouteInfo.kernel_host`/`kernel_port` come from that kernel.
- Per-kernel `shutdown_priority` so gateway sidecars get SIGTERM first (drain), then `main` + helpers.

Unlocks: co-located guardrail / cache / billing sidecar pattern. Most in-house use cases.

### Phase 2 — Internal-only circuits

Adds a `CircuitVisibility` enum (`PUBLIC` | `INTERNAL`) on `Circuit` and a matching `EndpointVisibility` on `EndpointRow`. `open_to_public: bool` becomes a deprecated alias for `visibility=PUBLIC + requires_token`, removed after one release.

Worker behaviour for `INTERNAL` circuits:

- No registration on the public port-mode or wildcard frontend.
- New internal frontend bound on an operator-configured cluster-private interface (config under `proxy_worker.internal_proxy`).
- `Circuit.get_endpoint_url()` gains an `INTERNAL` arm yielding a cluster-local URL.

Internal circuits still honour `EndpointToken` JWT checks by default; `requires_token=False` is allowed for tightly-coupled upstream/downstream pairs.

Unlocks: true private endpoints — network isolation, not just auth gating.

### Phase 3 — `upstream_endpoint_id`

A thin relational pointer on `EndpointRow`:

```python
upstream_endpoint_id: Mapped[UUID | None] = mapped_column(
GUID, ForeignKey("endpoints.id", ondelete="RESTRICT"), nullable=True,
)
```

When set, the manager:

1. Auto-provisions an `EndpointTokenRow` for the upstream scoped to the downstream's session owner, and injects `BACKENDAI_UPSTREAM_URL` / `BACKENDAI_UPSTREAM_TOKEN` into the downstream's kernels. Token rotation on a configurable cadence.
1. Enforces lifecycle coupling: upstream cannot be deleted while bound (FK + manager precheck with a clear error).
1. Exposes `upstream` / `downstreams` edges in the GraphQL `Endpoint` resolver.
1. Emits audit log entries on bind / unbind.

Deliberately **not** done: no request forwarding by the manager, no traffic policy. The downstream's gateway sidecar makes the actual HTTP call.

Unlocks: shared private guardrail / cache pattern end-to-end with no external orchestration.

## Concrete patterns this enables

### Pattern A — co-located guardrail (Phase 1 only)

```yaml
service:
start_command: ["python", "serve.py"] # main = model
port: 9000
sidecars:
- name: guardrail
image: /safety/llamaguard-gw:1.0
start_command: ["python", "gateway.py"]
ports: [8000]
role: gateway
environ:
UPSTREAM_URL: "http://main1:9000"
```

### Pattern B — shared private guardrail (Phases 1 + 2 + 3)

```yaml
# Endpoint U (private, shared)
name: llamaguard-shared
visibility: internal
service: { ... LlamaGuard runtime ... }

# Endpoint D (public gateway, per model)
name: llama-3-70b
upstream_endpoint_id:
service:
start_command: ["python", "serve.py"]
port: 9000
sidecars:
- name: gateway
image: /safety/guardrail-gw:1.0
role: gateway
ports: [8000]
# BACKENDAI_UPSTREAM_{URL,TOKEN} injected by the manager
```

## Migration & compatibility

- All three phases are **additive**. Endpoints with `cluster_size=1`, `visibility=PUBLIC`, `upstream_endpoint_id=NULL` behave identically to today.
- `open_to_public: bool` aliased for one release cycle, then removed.
- Migration defaults `visibility=PUBLIC`; no data change.
- `cluster_mode=MULTI_NODE` with non-empty `sidecars` is rejected at `Service.create` until a future phase 4.

## Alternatives considered

- **Extend** `pre_start_actions`. Rejected: actions run inside the main container's process tree — no separate network identity, image, resource slots, or failure domain.
- **AppProxy HTTP middleware chain.** Rejected: heavy guardrails are themselves LLMs (not in-proxy material), and middleware forces protocol-awareness into AppProxy. Operators wanting lightweight L7 filters can still use the Traefik backend's middleware.
- **First-class** `EndpointKind = Model | Guardrail | Router`. Rejected: bakes one use case into the schema; within a year we'd be adding `Cache`, `Mirror`, `Ensemble`, etc.
- **Service mesh (Istio/Linkerd) sidecar injection.** Rejected: many target environments are not K8s, and mesh-level injection doesn't see per-session bridge networks.

## Open questions

1. **Frontend kernel selection in current code.** Which kernel's `kernel_host:kernel_port` is registered as the circuit backend today? The route/circuit registration path for multi-kernel sessions wasn't fully traced — phase 1 hinges on this being explicit. Maintainer input welcome.
1. **Sidecar declaration site.** `model-definition.yaml` (model-artifact level) vs. deployment spec. Proposal favours deployment spec; a `required_sidecars` on `ModelDefinition` could be a follow-up.
1. **Token rotation cadence for** `upstream_endpoint_id`. Default 24h? Aligned with the endpoint's own token policy? Per-upstream override?
1. **Internal DNS strategy.** Operator-run `*.internal.bai` resolver vs. a coordinator-served internal resolver API that rewrites URLs to IPs at binding time.
1. **Shutdown drain default.** Phase-1 proposes 5s for gateway drain; needs measurement on realistic guardrail payloads.
1. `MULTI_NODE` + sidecars. Deferred. Would require the overlay plugin to tolerate sidecar joining and the scheduler to co-locate sidecars with `main`'s node.
1. **Naming.** `upstream_endpoint_id` vs. `protects_endpoint_id` vs. `backing_endpoint_id`. Current pick is the most domain-neutral.

## Acceptance

This Epic is complete when:

- Phase 1, 2, and 3 child Tasks are merged.
- A reference gateway sidecar image (LlamaGuard-style) is documented as an example, not a platform dependency.
- Migration guide for `open_to_public` → `visibility` + `requires_token` is published.
- An end-to-end smoke test exercises Pattern B (private upstream + public gateway + injected token).

JIRA Issue: BA-5863