kubeflow / kubeflow/kale

feature(backend) Add CLI support to compile notebooks to native Kubernetes manifests for GitOps/CI/CD

Open
#799 6 comments 0 reactions 1 assignee Claimed by @Bhavd33p View on GitHub
2.x candidate area/backend good first issue
Dominant language
Python
Stars
705
Forks
161
Avg merge
6d 3h
Merged PRs (30d)
10

Description

### Feature Area

### What feature would you like to see?

Add first-class CLI support in Kale to compile a tagged Jupyter notebook (`.ipynb`) into **native Kubernetes manifests** for Kubeflow Pipelines (KFP “kubernetes” / manifest store), not only into KFP IR YAML (`.pipeline.yaml`) for upload via the KFP API.

Today, `kale --nb ` compiles to KFP v2 DSL (`.kale/.kale.py`) and then to a pipeline package YAML intended for `upload_pipeline` / `run_pipeline`. For GitOps and CI/CD we need a **compile-only** path that emits manifests we can version, review, and apply with standard cluster tooling.

Proposed CLI behavior (example):

```bash
kale --nb pipelines/train.ipynb \
--kubernetes-manifest-format \
--kubernetes-namespace kubeflow \
--pipeline_name weekly-churn \
--pipeline-display-name "Weekly churn training" \
--pipeline-version-name weekly-churn-v1 \
--no-include-pipeline-manifest # optional: emit only PipelineVersion + workloads
```

Expected outcome:

- Kale invokes the KFP SDK compiler with `kubernetes_manifest_format=True` and `KubernetesManifestOptions` (namespace, pipeline/version names, whether to include the `Pipeline` CR).
- Output is written next to the generated DSL, e.g. `.kale/.pipeline.k8s.yaml` (or a documented, overridable output path).
- `--upload_pipeline` / `--run_pipeline` are rejected in this mode with a clear message (deploy via `kubectl apply -f` or a GitOps controller instead).
- User guide and a small samples snippet show a minimal GitLab/GitHub Actions job: install Kale → compile notebook → commit/apply manifest.

Optional follow-ups (can be separate issues): `--output `for artifact path, stdout-only mode for CI artifacts, and parity in the JupyterLab RPC path (`compile_into_native`) so UI and CLI behave the same.

### What is the use case or pain point?

Our team promotes notebooks from development to production through CI/CD, not by clicking “Run” in Jupyter or calling the KFP upload API from laptops.

We want:

1. Reproducible builds — The pipeline definition is a manifest in git, built from the same `.ipynb` the data scientists and data analytics authored.
2. GitOps-friendly deploys — Argo CD / Flux / `kubectl apply` apply `Pipeline`, `PipelineVersion`, and related resources to the cluster.
3. No runtime dependency on KFP upload from CI — CI agents often lack cluster credentials for `kfp.Client.upload_pipeline`, but can compile offline and push manifests to the repo or cluster.
4. Alignment with KFP kubernetes store — We run KFP with the Kubernetes-native pipeline store; manifests are the source of truth, not tarball uploads to the API server.

Without CLI support, Kale is optimized for interactive compile-and-upload workflows, which does not fit our production promotion model.

### Is there a workaround currently?

Partial workarounds, all heavier than a single Kale command:

1. Manual two-step compile
- kale --nb notebook.ipynb to generate `.kale/.kale.py`.
- Hand-write or maintain a small script that imports the generated DSL and calls `kfp.compiler.Compiler().compile(..., kubernetes_manifest_format=True, kubernetes_manifest_options=...)`.
- Fragile: every notebook author or CI job must duplicate KFP compiler options Kale already knows (pipeline name, namespace, version naming).

2. Compile IR YAML only, convert elsewhere
- Use default Kale output (`.pipeline.yaml`) and rely on cluster-side tooling or custom scripts to translate IR → manifests. Extra tooling and not always equivalent to KFP’s native manifest compiler path.

3. Jupyter-only / API upload path
- Use Kale Lab extension or `kale --run_pipeline --kfp_host ...` from a trusted environment. Does not give git-reviewed manifests and couples deploy to KFP API availability and credentials.

4. Fork Kale
- Patch `kale/cli.py` and `kale/common/kfputils.py` to pass `kubernetes_manifest_format` through. Works but is costly to maintain across Kale/KFP upgrades.

---

Love this idea? Give it a 👍.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.