[RFC]: AIBrix console deployment Provider Abstraction with Kubernetes Implementation

### Summary

This RFC proposes a standardized deployment provider abstraction for AIBrix Console.

The change makes Console deployments template-driven and provider-backed. Users create deployments by selecting a model deployment template, choosing an implementation kind such as `k8s-deployment`, and optionally applying runtime overrides.

The first implementation targets Kubernetes and maps Console deployments to native Kubernetes `Deployment`, `Service`, and optional `HorizontalPodAutoscaler` resources.

### Motivation

AIBrix Console needs a stable deployment abstraction that can support multiple runtime backends without coupling the UI or API handlers to a specific infrastructure implementation.

The current deployment flow needs clearer separation between deployment intent, runtime implementation, and runtime status observation. Standardizing this boundary allows Console to reuse model deployment templates, route deployments to different providers, and evolve toward richer reconciliation and status management.

This proposal improves extensibility by introducing a provider contract while keeping Kubernetes as the first concrete implementation.

### Proposed Change

## Overview

This RFC proposes a provider-based deployment architecture for AIBrix Console.

The design introduces:

- A template-driven deployment creation flow
- A standardized deployment provider lifecycle interface
- A provider registry keyed by `implementation.kind`
- A first Kubernetes provider implementation
- Clearer read semantics for deployment list and detail APIs

The goal is to make `Deployment` a stable Console product abstraction while allowing different runtime backends to implement their own provisioning and lifecycle behavior.

## Current Scope

This proposal covers the first implementation phase.

Included:

- Create deployment from `ModelDeploymentTemplate`
- Select deployment implementation through `implementation.kind`
- Support `k8s-deployment` as the first provider
- Create native Kubernetes `Deployment`, `Service`, and optional `HPA`
- Support provider-aware deployment detail reads
- Keep deployment list reads lightweight by returning persisted snapshots
- Delete Kubernetes runtime resources when deleting Console deployments

Not included in this phase:

- Full background reconciliation loop
- Rich typed deployment status object
- Deployment list pagination
- Log, event, and metrics APIs
- Full `stormservice` provider implementation
- Image pull secret abstraction
- Full RBAC or secret policy standardization

## Architecture

The proposed architecture has three layers:

1. Template Layer
- Describes model-serving intent and recommended runtime defaults.
- Provides engine, accelerator, scaling, topology, and compatibility metadata.

2. Provider Layer
- Owns concrete runtime lifecycle operations.
- Implements create, get, list, update, and delete against a specific backend.

3. Console API Layer
- Resolves templates.
- Selects providers.
- Persists deployment metadata.
- Shapes API responses for frontend consumption.

```mermaid
flowchart LR
UI[Console Web UI] --> API[DeploymentService]
API --> STORE[(Console Store)]
API --> TPL[Template Lookup]
API --> REG[Provider Registry]

TPL --> TMPL[ModelDeploymentTemplate]
REG --> K8S[Kubernetes Provider deployment/aibrix stormservice/etc..]
REG --> FUTURE[Future Providers]

K8S --> KAPI[Kubernetes API Server]
KAPI --> DEPLOY[Deployment]
KAPI --> SVC[Service]
KAPI --> HPA[HorizontalPodAutoscaler]
```
## Frontend Changes

The `Create Deployment` flow is updated to support the new template-driven and provider-backed deployment model. Users now explicitly select a base model, an active deployment template, and a deployment implementation kind such as `k8s-deployment`. After template selection, the frontend applies template-derived defaults such as accelerator settings, quantization, and scaling configuration, while still allowing runtime overrides. The frontend submits normalized `template`, `implementation`, and `overrides` fields to the backend, while keeping provider-specific runtime logic entirely on the server side.

## Deployment Request Model

Deployment creation is standardized around three concepts:

- `template`
- `implementation`
- `overrides`

Example request:

```json
{
"name": "gsm8k-math-sample",
"template": {
"model_id": "model-123",
"template_id": "template-456"
},
"implementation": {
"kind": "k8s-deployment"
},
"overrides": {
"region": "GLOBAL",
"min_replicas": 1,
"max_replicas": 2,
"enable_auto_scaling": true,
"enable_multi_lora": false
}
}
```

The deployment response stores provider traceability:

- `template_id`
- `template_version`
- `implementation_kind`

These fields make it possible to understand how a deployment was created and which provider owns its runtime lifecycle.

## Template Model Extensions

Model deployment templates are extended with lightweight metadata that keeps templates portable across providers.

Relevant concepts:

- `topology`
- Describes the high-level serving topology.
- `compatibility`
- Declares which provider implementation kinds are supported.
- `scaling_defaults`
- Provides recommended replica and autoscaling defaults.

Example:

```yaml
spec:
topology:
kind: standard
compatibility:
implementation_kinds:
- k8s-deployment
scaling_defaults:
min_replicas: 1
max_replicas: 2
enable_auto_scaling: true
```

This avoids embedding Kubernetes manifests directly in templates while still giving providers enough information to validate and instantiate a deployment.

## Provider Interface

The provider abstraction is intentionally CRUD-shaped.

```go
type DeploymentDriver interface {
Kind() string
Validate(ctx context.Context, template *pb.ModelDeploymentTemplate, req *pb.CreateDeploymentRequest) error
Create(ctx context.Context, template *pb.ModelDeploymentTemplate, req *pb.CreateDeploymentRequest) (*pb.Deployment, error)
Get(ctx context.Context, deployment *pb.Deployment) (*pb.Deployment, error)
List(ctx context.Context, deployments []*pb.Deployment) ([]*pb.Deployment, error)
Update(ctx context.Context, deployment *pb.Deployment) (*pb.Deployment, error)
Delete(ctx context.Context, deployment *pb.Deployment) error
}
```

Provider responsibilities:

- Validate whether a template can be deployed by this provider.
- Create runtime resources.
- Read live runtime state.
- Update runtime configuration where supported.
- Delete runtime resources.
- Return a normalized Console `Deployment` object.

## Provider Registry

A provider registry maps `implementation.kind` to a concrete provider.

```mermaid
flowchart TD
REQ[CreateDeploymentRequest] --> KIND[implementation.kind]
KIND --> REG[Provider Registry]
REG --> K8S[k8s-deployment]
REG --> FUTURE[future providers]
```

Initial provider:

- `k8s-deployment`

Future providers may include:

- `stormservice`
- cloud-managed serving runtimes
- external deployment backends

If `implementation.kind` is omitted, Console may default to `k8s-deployment` for compatibility.

## Kubernetes Provider

The first concrete provider is `k8s-deployment`.

It maps Console deployments to native Kubernetes resources.

Created resources:

- `apps/v1 Deployment`
- `core/v1 Service`
- optional `autoscaling/v2 HorizontalPodAutoscaler`

```mermaid
flowchart LR
TEMPLATE[ModelDeploymentTemplate] --> BUILD[Provider Build Logic]
OVERRIDES[DeploymentOverrides] --> BUILD
BUILD --> KDEPLOY[apps/v1 Deployment]
BUILD --> KSVC[v1 Service]
BUILD --> KHPA[autoscaling/v2 HPA]
```

### Kubernetes Validation

The Kubernetes provider validates:

- deployment template exists
- template spec exists
- deployment name exists
- declared compatibility includes `k8s-deployment`, when compatibility is present
- unsupported topology is rejected
- `template.spec.engine.image` is present

The `engine.image` field is required because Kubernetes pods require a concrete container image.

### Kubernetes Runtime Mapping

The provider maps template fields into Kubernetes runtime resources:

- container image from `spec.engine.image`
- serve args from `spec.engine.serve_args`
- model source flags from `spec.model_source`
- health probes from `spec.engine.health_endpoint`
- GPU requests and limits from `spec.accelerator`
- Hugging Face token from referenced Kubernetes secret when configured
- replica settings from template defaults plus request overrides
- HPA from autoscaling settings

## Kubernetes Configuration

The Kubernetes provider can connect to Kubernetes through explicit kubeconfig or in-cluster config.

Environment variables:

- `K8S_KUBECONFIG`
- `K8S_CONTEXT`
- `K8S_NAMESPACE`
- `K8S_SERVICE_TYPE`
- `K8S_CONTAINER_PORT`
- `K8S_SERVICE_PORT`
- `K8S_HPA_TARGET_CPU_UTILIZATION`

Example local configuration:

```bash
export K8S_KUBECONFIG=/tmp/kind-aibrix-console.kubeconfig
export K8S_CONTEXT=kind-aibrix-console
export K8S_NAMESPACE=aibrix-console
export K8S_SERVICE_TYPE=NodePort
export K8S_CONTAINER_PORT=8000
export K8S_SERVICE_PORT=8000
export K8S_HPA_TARGET_CPU_UTILIZATION=80
```

## Create Flow

```mermaid
sequenceDiagram
participant UI as Console UI
participant API as DeploymentService
participant Store as Store
participant Registry as Provider Registry
participant K8s as Kubernetes Provider
participant Kube as Kubernetes API

UI->>API: CreateDeployment(template, implementation, overrides)
API->>Store: GetModelDeploymentTemplate(model_id, template_id)
Store-->>API: template
API->>Registry: Get(implementation.kind)
Registry-->>API: provider
API->>K8s: Validate(template, request)
API->>K8s: Create(template, request)
K8s->>Kube: Create Namespace/Deployment/Service/HPA
Kube-->>K8s: success
K8s-->>API: Deployment(runtime metadata)
API->>Store: SaveDeployment(...)
Store-->>API: persisted deployment
API-->>UI: Deployment
```

## Read Semantics

This proposal separates list behavior from detail behavior.

### List Deployments

`ListDeployments` returns persisted store snapshots.

It should not synchronously fan out to live provider reads for each row.

Reasons:

- list APIs should remain lightweight
- per-row provider reads create N+1 runtime calls
- list latency should not depend on provider health for every row
- runtime-heavy reads belong in detail or reconciliation paths

### Get Deployment

`GetDeployment` performs provider-aware read-through.

Flow:

1. Read deployment metadata from store.
2. Resolve provider from `implementation_kind`.
3. Fetch live runtime state from provider.
4. Persist refreshed snapshot back to store.
5. Return the refreshed deployment.

```mermaid
flowchart TD
LIST[ListDeployments] --> STORE1[Read Store Snapshot]
DETAIL[GetDeployment] --> STORE2[Read Store Record]
STORE2 --> PROVIDER[Provider Get]
PROVIDER --> SNAPSHOT[Refresh Store Snapshot]
SNAPSHOT --> RETURN[Return Deployment]
```

This keeps list cheap while still allowing the detail page to display live runtime state.

## Update Semantics

The provider interface includes `Update`.

For Kubernetes, the initial update behavior supports:

- updating replica count
- creating, updating, or removing HPA based on autoscaling configuration

The public API does not need to expose all update behavior in the first phase. The backend interface is prepared for future `UpdateDeployment` support.

## Delete Semantics

Deletion follows this sequence:

1. Read deployment from store.
2. Resolve provider from `implementation_kind`.
3. Delete provider runtime resources.
4. Delete the Console store record.

For Kubernetes, provider deletion removes:

- `HorizontalPodAutoscaler`
- `Service`
- `Deployment`

## Frontend Behavior

The frontend create flow should explicitly capture:

- selected model
- selected deployment template
- selected implementation kind
- runtime overrides

```mermaid
flowchart LR
MODEL[Select Model] --> TEMPLATE[Select Template]
TEMPLATE --> IMPL[Select Implementation]
IMPL --> OVERRIDE[Adjust Overrides]
OVERRIDE --> CREATE[Create Deployment]
```

The frontend does not need to know provider-specific runtime details. It submits normalized API fields and renders normalized deployment responses.

List and detail behavior should follow the backend contract:

- list page reads snapshot data
- detail page reads provider-refreshed deployment data
- future background reconciliation can make list snapshots fresher without changing frontend semantics

## Status Model

The current implementation keeps deployment status as a string for compatibility.

Initial phases may include:

- `Deploying`
- `Ready`
- `Scaling`
- `Degraded`
- `Failed`
- `Deleted`

The Kubernetes provider derives status from:

- deployment existence
- observed generation
- ready replicas
- available replicas
- deployment conditions
- service existence
- HPA existence

A future phase should introduce structured status:

```proto
message DeploymentStatus {
string phase = 1;
string reason = 2;
string message = 3;
int32 desired_replicas = 4;
int32 ready_replicas = 5;
int32 available_replicas = 6;
string observed_at = 7;
}
```

This is intentionally deferred to keep the first provider abstraction focused.

## Compatibility

This proposal is incremental.

Compatibility is preserved by:

- keeping existing flat deployment fields in `CreateDeploymentRequest`
- adding template-driven fields without removing old fields
- defaulting the implementation kind when omitted
- continuing to use the Console store as the source of persisted deployment metadata

New deployments can use the provider-backed path, while existing store-backed behavior remains compatible.

## Security Considerations

Security-sensitive areas include:

- kubeconfig or in-cluster credentials
- namespace access
- Kubernetes secret references
- Hugging Face token projection
- future image pull secret handling
- provider RBAC permissions

This proposal does not yet define a complete secret or RBAC policy. It introduces the provider boundary where those policies can be standardized later.

## Operational Considerations

Potential failure modes:

- invalid template
- missing container image
- unsupported topology
- Kubernetes API server unavailable
- image pull failure
- pod readiness failure
- HPA metrics unavailable
- insufficient cluster resources

The first implementation surfaces provider errors through the deployment API. Future work should improve this with structured status reasons and event reporting.

## Alternatives Considered

### Hardcode Kubernetes in API handlers

Rejected because it couples the Console API directly to Kubernetes and makes future providers difficult.

### Store Kubernetes manifests inside templates

Rejected because it makes templates provider-specific and reduces portability.

### Make list perform live provider reads

Rejected for the standard path because it creates N+1 runtime calls and makes list latency depend on provider health.

## Rollout Plan

### Phase 1: Provider Abstraction and Kubernetes Implementation

- Add template, implementation, and override fields to deployment creation.
- Add provider registry.
- Add Kubernetes provider.
- Create Kubernetes `Deployment`, `Service`, and optional `HPA`.
- Keep list snapshot-based.
- Make detail provider-aware.

### Phase 2: Reconciliation and Status

- Add background snapshot reconciliation.
- Refresh non-terminal deployments periodically.
- Introduce structured deployment status.
- Add status reason and message.

### Phase 3: Multi-Provider Expansion

- Add `stormservice` provider.
- Normalize provider-specific updates.
- Add provider-specific runtime details through stable API fields.

## Example kind Testing Flow

```bash
kind create cluster --name aibrix-console
kind get kubeconfig --name aibrix-console > /tmp/kind-aibrix-console.kubeconfig

export K8S_KUBECONFIG=/tmp/kind-aibrix-console.kubeconfig
export K8S_CONTEXT=kind-aibrix-console
export K8S_NAMESPACE=aibrix-console
export K8S_SERVICE_TYPE=NodePort
export K8S_CONTAINER_PORT=8000
export K8S_SERVICE_PORT=8000
export K8S_HPA_TARGET_CPU_UTILIZATION=80
```

After starting Console, create a deployment from the UI and verify:

```bash
kubectl --kubeconfig /tmp/kind-aibrix-console.kubeconfig \
--context kind-aibrix-console \
-n aibrix-console \
get deploy,svc,hpa,pods
```

## Conclusion

This proposal standardizes Console deployments around a provider abstraction.

It keeps the first implementation focused and practical:

- template-driven deployment intent
- provider-owned runtime lifecycle
- Kubernetes as the first backend
- lightweight list reads
- provider-aware detail reads

This establishes a foundation for background reconciliation, structured status, and additional deployment providers.

### Alternatives Considered

## Open Questions

- When should `UpdateDeployment` be exposed publicly?
- The status machine is remaining to be supply and define for deployment.
- Should provider-specific runtime details be embedded in `Deployment` or returned through a separate runtime detail object?
- How should image pull secrets be standardized across providers?
- How should provider-level events and logs/metrics be exposed?