Feast Control Plane UI

## Summary

Feast has a read-only UI that visualizes registry state. This proposal outlines evolving it into an interactive control plane where users can browse, edit, validate, and apply changes visually — and where AI agents can propose changes that humans verify on screen before applying.

This builds on the Feast UI (#2353), the REST API migration (#5411), CRUD requests (#5301, #5533, #5443), the context engine vision (#5761), and the existing MCP server integration.

## Problem

**The CLI + YAML workflow is powerful but exclusionary.**

Not every Feast user is a data engineer comfortable writing Python definitions. Data scientists and ML engineers who consume features often learn better through visualization than reading YAML. Today, creating a feature view requires knowing table names, column names, and types ahead of time — context-switching between a database client and the feature repo. The edit → `feast apply` → read terminal → fix → repeat loop is slow, and `feast plan` (dry-run) only works on `local` + `sqlite` (`_should_use_plan()` in `feature_store.py`). In managed environments (OpenShift, air-gapped clusters), terminal-inside-pod is often the only option — and those changes don't survive pod restarts.

**The existing UI shows state but cannot change it.**

The React UI (`ui/`) has pages for data sources, entities, feature views, feature services, permissions, and lineage. But:

- **Entirely read-only.** Zero `POST`/`PUT`/`DELETE` calls or `useMutation` hooks in the React app. `useLoadRegistry` fetches a single protobuf blob from `/registry` and derives everything from that snapshot.
- **REST registry API is GET-only.** All routes in `api/registry/rest/` are `@router.get`. The gRPC registry server has `ApplyEntity`, `ApplyFeatureView`, `ApplyPermission`, etc. — but these are not exposed via REST.
- **Lineage is visual but static.** The React Flow graph renders relationships but cannot be edited.
- **Permissions page says "manage" but only displays.** Fixing a group-based access mistake requires: edit Python → commit → `feast apply` → verify — 5-6 terminal steps for what should be one click.

**No verification surface for agentic workflows.**

Feast already has an MCP server (`fastapi_mcp` at `/mcp`) and OpenLineage integration. But without a visual layer, agents are black boxes — users can't see what an agent proposes, verify its choices, or catch mistakes before they reach the registry. Trust requires transparency, and transparency requires a screen.

## Proposed Solution

An interactive **Feast Control Plane UI** — not replacing CLI/YAML, but complementing it. Phased approach:

### Phase 1 — Live Registry + Basic Mutations (MVP)

- **Migrate UI from protobuf dump to REST API.** The REST GET endpoints already exist with pagination, sorting, filtering, and relationship inclusion — the UI just doesn't consume them yet (#5411).
- **Add REST write endpoints** wrapping the existing gRPC `Apply*`/`Delete*` RPCs.
- **Add `/plan` endpoint** calling `FeatureStore.plan()` for structured diff output (currently gated to local+sqlite — registry diffs don't depend on provider and could be generalized).
- **Create/edit forms** for entities, data sources, feature views with inline validation.
- **Preview → Apply workflow** with diff view and real-time status.

### Phase 2 — Schema Discovery + Interactive Lineage + Permissions

- **Schema discovery:** browse data source tables/columns, create feature views by selecting columns interactively (eliminates the database-client context-switch).
- **Interactive lineage:** click nodes to edit definitions inline; build on existing React Flow graph + permission overlays (`permissionUtils.ts`).
- **Permissions editor:** transform the display-only page into a real editor — select resource types, actions, policy type, save. Run `feast permissions check` logic to surface coverage gaps.

### Phase 3 — Agentic Workflows + Observability

- **Agent proposals as pending diffs:** an AI agent (via MCP) proposes a feature view → it appears in the UI as a visual diff → user reviews, modifies, or rejects → approved changes go through the standard validated apply path.
- **Agent-assisted debugging:** agent analyzes materialization failures or schema drift, surfaces findings in UI with suggested fixes.
- **Observability:** feature freshness, materialization history, data source health, usage metrics. Builds on #5920 and existing OpenLineage emitter in `FeatureStore`.

The control plane becomes the **shared workspace** where human intent and agent execution meet — the agent proposes, the UI displays, the human decides.

## Current State vs What Is Needed

| Capability | Today | Needed |
|------------|-------|--------|
| Registry browsing | Protobuf dump via `useLoadRegistry` | Migrate to existing REST API endpoints |
| Registry mutations | gRPC `Apply*/Delete*`; REST is GET-only | REST write wrappers |
| Apply via HTTP | None — only `FeatureStore.apply()` Python API | REST `/apply` calling `FeatureStore.apply()` |
| Plan / diff | `FeatureStore.plan()` — gated to local+sqlite | REST `/plan`; lift provider gate for registry diffs |
| Schema discovery | `provider.validate_data_source()` for validation | New endpoint to enumerate tables/columns |
| Lineage | React Flow graph, read-only | Editable nodes, click-to-edit definitions |
| Permissions | Display-only page | Write API + edit forms |
| MCP / agentic | `fastapi_mcp` at `/mcp` (serving endpoints only) | Registry mutation tools + approval workflow |

## Technical Notes

- **Backend:** REST write layer must call `FeatureStore.apply()` / `FeatureStore.plan()` — not bypass to raw gRPC Apply (which skips validation, inference, and infra updates). `plan()` already accepts `desired_repo_contents: RepoContents` — the entry point for UI-driven diffs without generating Python files.
- **Frontend:** Extend existing React UI (Elastic UI, React Query, React Flow). Add mutation hooks (currently zero).
- **Safety:** All mutations go through `FeatureStore` validation. Destructive ops require confirmation. Diff preview before apply.
- **Auth:** Reuse existing OIDC + K8s auth. UI authenticates through the same mechanism as all other Feast clients.

## Alternatives Considered

- **CLI/YAML only.** Works for experienced engineers. Does not address discoverability, onboarding, or agentic verification.
- **Internal tools instead of upstream.** Fragments the ecosystem. Community demand (#5301, #5533, #5443) shows this is a shared need.
- **Wait for agentic maturity.** MCP and OpenLineage already exist. Building the control plane now provides the human-in-the-loop layer that responsible agent workflows need.

## Open Questions

1. **Where should this live?** Extend `ui/` (simplest), separate repo, or plugin?
2. **Write API: gRPC wrappers vs full `FeatureStore.apply()` path?** The latter is safer but requires constructing `RepoContents` from API input.
3. **Should `FeatureStore.plan()` be generalized?** Registry diffs don't depend on provider — only infra diffs do.
4. **Code-vs-registry source of truth?** If teams use Git, UI-only edits could diverge. Should the UI generate exportable definitions?
5. **How far should MCP go?** Direct mutation tools, or approval-gated proposals only?

## References

- #2353 — Feast Web UI
- #5411 — Migrate UI to REST API
- #5301, #5533, #5443 — CRUD UI requests
- #5920 — Feature Server Observability
- #5761 — Feast as Context Engine for AI Agents

## Authors
Aniket Paluskar, Cursor (Claude)