Pausing Kubeflow notebooks

As discussed in the Notebooks Community call, I'll try to state my usecase / problem here and we can see whether this is something that can be added to the list of planned usecase or whether there is an easy fix that we could deploy for our installation of Kubeflow.

In any case I'm happy to work on documentation, a more detailed user story, test possible implementations or do simple developments on my own.

---

There should be a non-destructive way to pause & resume Kubeflow Notebooks / Workspaces in order to perform any of the following action as a cluster admin:

- Upgrade the underlying kubernetes cluster
- Save expensive resources by culling inactive notebooks
- Save expensive resources by evicting/pausing notebook pods from expensive nodes (already mentioned in https://github.com/kubeflow/notebooks/issues/307)

To my understanding this is currently not possible without losing the temporary state that the AI Engineer Persona might have created in the meantime, for example developing a new base image for using within Kubeflow Pipelines or running a (potentially costly) Machine Learning experiment that would need to be done in case the pod is restarted since the Notebook kernel state is lost.

Since this feature is currently missing the resources in our Kubeflow installation feel way more static on the infrastructure side (outside of announced downtime windows) than they would ideally be due to fear of disrupting the ML Engineers work and forcing duplicate work and resulting frustration.

Start by reviewing the requested pause and resume use cases and the related issue #307. Investigate how Kubeflow Notebooks currently handles notebook pods and temporary kernel state. Done means defining and validating a non-destructive way for cluster administrators to pause, resume, cull, or evict notebooks without losing work.

Written by the indexing model from the issue text.