claimed-framework / claimed-framework/mlx_deprecated
Replace `kfp-notebook`/`elyra-server` with a KFP component to run notebooks
- Dominant language
- Python
- Stars
- 220
- Forks
- 54
- PR merge metrics
- No merged PRs in 30d
Description
**Current State:**
We currently depend on the `elyra-server` package with the `ExecuteFileOp` to run notebooks in a Kubeflow Pipeline. While that works fairly well and provides much flexibility, there are some drawbacks.
**The Problem:**
One of the biggest drawbacks are the Python packages that are required by `elyra`, most of which are not essential to generate the pipeline code to run the notebook. Note, that the `papermill` package with its dependencies on `ipython` etc are only required on the Pod running the notebook, not on the MLX API server pod that generates the pipeline to run the notebook. Many of the remaining packages required by `elyra` overlap with the packages required by `kfp`/`kfp-tekton`, sometimes requiring conflicting and occasionally irreconcilable versions.
**Proposed Solution:**
Instead of using Elyra to generate the pipelines to run notebooks, we could use a straight-forward KFP component, as we have done in the past. An example of a notebook runner component with sample pipeline exists in KFP:
- [https://github.com/kubeflow/pipelines/[…]/contrib/notebooks/Run_notebook_using_papermill/component.yaml](https://github.com/kubeflow/pipelines/blob/74c7773ca40decfd0d4ed40dc93a6af591bbc190/components/contrib/notebooks/Run_notebook_using_papermill/component.yaml)
- [https://github.com/kubeflow/pipelines/[...]/components/contrib/notebooks/samples/sample_pipeline.py](https://github.com/kubeflow/pipelines/blob/74c7773ca40decfd0d4ed40dc93a6af591bbc190/components/contrib/notebooks/samples/sample_pipeline.py)
The Elyra team had considered switching to a KFP component, but they have not done any further investigation. Publishing a component -- that is married to a base Docker image -- would probably not meet the needs of most Elyra users (@ptitzler)
Having the flexibility to specify the Docker image to be used to run the notebook is one important consideration. Since MLX uses [code templates to generate sample pipelines to launch notebooks](https://github.com/machine-learning-exchange/mlx/blob/9fd261248b1dc6fe9bcff55fbfef678eb25b3444/api/server/swagger_server/code_templates/run_notebook.TEMPLATE.py#L25-L34), we could utilize [`kfp.components.create_component_from_func()`](https://www.kubeflow.org/docs/components/pipelines/sdk/python-function-components/#getting-started-with-python-function-based-components) which takes a `base_image` argument:
```Python
add_op = create_component_from_func(
func=add,
base_image='python:3.7', # Optional
output_component_file='add.component.yaml', # Optional
packages_to_install=['pandas==0.24'], # Optional
)
```
Or we could use `kfp.components.create_component_from_file` and replace the image attribute in the `ComponentSpec` > .. > `ContainerSpec`. Or, even more crudely, we could use `kfp.components.create_component_from_text` and replace an `${image}` placeholder in a `component.yaml.template`.
**Related Issues:**
- #300 Create Docker image(s) to run Notebooks
- #299 MLX API build broken
**Interested Parties:**
- **Elyra** team: @ptitzler @akchinSTC
- **MLX** team: @ckadner @Tomcli @animeshsingh
Contributor guide
Assessment
This issue has not been assessed yet.