feature(backend) Add CLI support to compile notebooks to native Kubernetes manifests for GitOps/CI/CD
- Dominant language
- Python
- Stars
- 705
- Forks
- 161
- Avg merge
- 6d 3h
- Merged PRs (30d)
- 10
Description
### Feature Area
### What feature would you like to see?
Add first-class CLI support in Kale to compile a tagged Jupyter notebook (`.ipynb`) into **native Kubernetes manifests** for Kubeflow Pipelines (KFP “kubernetes” / manifest store), not only into KFP IR YAML (`.pipeline.yaml`) for upload via the KFP API.
Today, `kale --nb ` compiles to KFP v2 DSL (`.kale/.kale.py`) and then to a pipeline package YAML intended for `upload_pipeline` / `run_pipeline`. For GitOps and CI/CD we need a **compile-only** path that emits manifests we can version, review, and apply with standard cluster tooling.
Proposed CLI behavior (example):
```bash
kale --nb pipelines/train.ipynb \
--kubernetes-manifest-format \
--kubernetes-namespace kubeflow \
--pipeline_name weekly-churn \
--pipeline-display-name "Weekly churn training" \
--pipeline-version-name weekly-churn-v1 \
--no-include-pipeline-manifest # optional: emit only PipelineVersion + workloads
```
Expected outcome:
- Kale invokes the KFP SDK compiler with `kubernetes_manifest_format=True` and `KubernetesManifestOptions` (namespace, pipeline/version names, whether to include the `Pipeline` CR).
- Output is written next to the generated DSL, e.g. `.kale/.pipeline.k8s.yaml` (or a documented, overridable output path).
- `--upload_pipeline` / `--run_pipeline` are rejected in this mode with a clear message (deploy via `kubectl apply -f` or a GitOps controller instead).
- User guide and a small samples snippet show a minimal GitLab/GitHub Actions job: install Kale → compile notebook → commit/apply manifest.
Optional follow-ups (can be separate issues): `--output `for artifact path, stdout-only mode for CI artifacts, and parity in the JupyterLab RPC path (`compile_into_native`) so UI and CLI behave the same.
### What is the use case or pain point?
Our team promotes notebooks from development to production through CI/CD, not by clicking “Run” in Jupyter or calling the KFP upload API from laptops.
We want:
1. Reproducible builds — The pipeline definition is a manifest in git, built from the same `.ipynb` the data scientists and data analytics authored.
2. GitOps-friendly deploys — Argo CD / Flux / `kubectl apply` apply `Pipeline`, `PipelineVersion`, and related resources to the cluster.
3. No runtime dependency on KFP upload from CI — CI agents often lack cluster credentials for `kfp.Client.upload_pipeline`, but can compile offline and push manifests to the repo or cluster.
4. Alignment with KFP kubernetes store — We run KFP with the Kubernetes-native pipeline store; manifests are the source of truth, not tarball uploads to the API server.
Without CLI support, Kale is optimized for interactive compile-and-upload workflows, which does not fit our production promotion model.
### Is there a workaround currently?
Partial workarounds, all heavier than a single Kale command:
1. Manual two-step compile
- kale --nb notebook.ipynb to generate `.kale/.kale.py`.
- Hand-write or maintain a small script that imports the generated DSL and calls `kfp.compiler.Compiler().compile(..., kubernetes_manifest_format=True, kubernetes_manifest_options=...)`.
- Fragile: every notebook author or CI job must duplicate KFP compiler options Kale already knows (pipeline name, namespace, version naming).
2. Compile IR YAML only, convert elsewhere
- Use default Kale output (`.pipeline.yaml`) and rely on cluster-side tooling or custom scripts to translate IR → manifests. Extra tooling and not always equivalent to KFP’s native manifest compiler path.
3. Jupyter-only / API upload path
- Use Kale Lab extension or `kale --run_pipeline --kfp_host ...` from a trusted environment. Does not give git-reviewed manifests and couples deploy to KFP API availability and credentials.
4. Fork Kale
- Patch `kale/cli.py` and `kale/common/kfputils.py` to pass `kubernetes_manifest_format` through. Works but is costly to maintain across Kale/KFP upgrades.
---
Love this idea? Give it a 👍.
Contributor guide
Assessment
This issue has not been assessed yet.