Kubeflow Notebooks 2.0 (Kubeflow Workspaces)

/kind feature

Here is my proposal for Kubeflow Notebooks 2.0, which can be called "Kubeflow Workspaces".

## Frequently Asked Questions

---
> ### What is Notebooks 2.0?
>
> Kubeflow Notebook 2.0 is the next evolution of Kubeflow Notebooks.
> It makes running IDEs on Kubernetes for ML/AI significantly better for users and cluster admins alike.

---

> ### Where can I find Notebooks 2.0?
>
> We are developing Notebooks 2.0 in [`kubeflow/notebooks` on the `notebooks-v2` branch](https://github.com/kubeflow/notebooks/tree/notebooks-v2).
>
> Here are direct links to each component's code:
>
> - [`./workspaces/controller/`](https://github.com/kubeflow/notebooks/tree/notebooks-v2/workspaces/controller): new Kubernetes controller (Golang, Kubebuilder)
> - [`./workspaces/backend/`](https://github.com/kubeflow/notebooks/tree/notebooks-v2/workspaces/backend): new REST API "backend for the frontend" (Golang)
> - [`./workspaces/frontend/`](https://github.com/kubeflow/notebooks/tree/notebooks-v2/workspaces/frontend): new frontend UI (NodeJS, React)

---

> ### How can I contribute to Notebooks 2.0?
>
> We welcome code contributions and ideas! The best way to get involved is participating on the [`#kubeflow-notebooks` Slack channel](https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels), and joining our weekly [notebooks community calls](https://docs.google.com/document/d/1SiWLah-U07hAc47sSsoI8-NkbkLnHl1YVzxw193QUKE/edit?tab=t.0) (currently at 8AM PT each Thursday).
>
> If you don't see any unassigned tasks, take a look at the current state of the code and make some suggestions (by creating issues on the `kubeflow/notebooks` repo or starting a thread in Slack).

---

> ### When will Notebooks 2.0 be ready?
>
> We hope to ship a widely available alpha version as an optional component of Kubeflow 1.11 (ETA mid 2025).
>
> However, we will release a testing version as soon as we have an end-to-end working platform (ETA late 2025), __subscribe to this issue__ for updates and to help us test.

---

# Motivation

The main idea has always been to make the Notebook CRD not just a wrapper around PodSpec, with the goal of abstracting away the Kubernetes resources from end users, while also giving cluster admins the ability to define a set of "templates" that end users can choose from.

The main benefits of this approach are:

- end-users can create Workspaces without needing to know anything about Kubernetes, because the UI literally becomes 3 dropdowns and 2 volume mounts:
- Choose a WorkspaceKind (e.g. "jupyter-lab", "vs-code", "rstudio")
- Choose an image from the approved list
- Choose a pod-config (e.g. "small_cpu", "big_gpu")
- Create/Mount a home volume (optional)
- Create/Mount data volumes (optional)
- cluster-admins can update the definitions of WorkspaceKinds without breaking existing Workspaces:
- they simply add a new image/pod-config option, and redirect the old one to the new one
- we can even provide a config to make the controller wait for a Workspace to restart before applying the redirect
(we can also make the Spawner UI display a warning that the Workspace needs to be restarted to get the new config)
- scheduling Pods with GPU resources often requires a number of Pod configs to be correctly aligned (e.g. tolerating a taint, and setting a resource limit), collecting all these configs into a single "pod-config" allows cluster-admins to provide drop-down options, rather than requiring users to understand the structure of their Kubernetes cluster.
- resizing an existing notebook is effectively impossible right now, because all the nice "spawner configs" are only available at the time you create the Notebook, once it's spawned you have a PodSpec.
- With WorkspaceKinds, we can make an "edit" button on the UI which allows you to pick from the current "pod-configs" and "images" available for the WorkspaceKind.

# Implementation

For more detailed information about the design, please see:

- [__Notebooks 2.0 - Design Document__](https://docs.google.com/document/d/1_zk06zebbaTBdJ8TdU07Ibky25hqHGARXjVcsp2qEnU/edit#heading=h.qt6x5supdfo4).

## New Components

- a new [__Workspace Controller__](https://github.com/kubeflow/notebooks/tree/notebooks-v2/workspaces/controller):
- Will manage and reconcile the new CRDs
- Will provide webhooks (for validation of CRD patches)
- Will be written in GoLang with Kubebuilder
- a new [__Workspace Backend API__](https://github.com/kubeflow/notebooks/tree/notebooks-v2/workspaces/backend)
- Will be the interface between the frontend and Kubernetes
- Will allow programmatic management of Notebooks, and allow easier replacement of frontend.
- Will be a REST API written in Golang
- a new [__Workspace Frontend__](https://github.com/kubeflow/notebooks/tree/notebooks-v2/workspaces/frontend):
- Will let users overview the workspaces in a namespace
- Will let users spawn, edit and connect to Workspaces
- Will be written in JS (React)

## New CRDs

The high-level overview is to split the Notebook CRD into these two CRDs:

- [`Workspace`](https://github.com/thesuperzapper/kubeflow-notebooks-v2-design/blob/main/crds/workspace.yaml) (namespaced resource)
- this is the resource that end-users create via the "Workspace Spawner UI" or `kubectl`
- it is NOT a wrapper around PodSpec
- [`WorkspaceKind`](https://github.com/thesuperzapper/kubeflow-notebooks-v2-design/blob/main/crds/workspace-kind.yaml) (cluster resource)
- this is the resource that cluster-admins create
- it specifies the template for a Workspace (e.g. "JupyterLab", "VSCode", "RStudio")
- initially, we would only support a "podTemplate" kind, which is very similar to the existing Notebook CRD,
but in the future, we could support other types of templates (e.g. "helmTemplate")

Start with the linked Notebooks 2.0 design document, then review the notebooks-v2 workspaces/controller, workspaces/backend, and workspaces/frontend directories. The proposal describes a broad new platform with Workspace and WorkspaceKind CRDs, but it does not define a bounded task or a specific completion criterion.

Written by the indexing model from the issue text.