feature(backend): Support KFP Pipeline Workspace for inter-step data passing via shared PVC
- Dominant language
- Python
- Stars
- 703
- Forks
- 161
- Avg merge
- 6d 3h
- Merged PRs (30d)
- 10
Description
## Summary
Kale currently passes data between pipeline steps using pickle-based serialization. This works for small objects but breaks down for large datasets, huge models, non-serializable Python objects (e.g. database connections), and ephemeral data stores.
KFP supports a workspace PVC model where steps read/write files via a shared filesystem, matching the notebook's mental model more closely.
A KEP will detail the architecture.
## Proposal
Support KFP's native workspace PVC as an alternative to pickle-based data passing between steps. Steps would share a filesystem path, enabling:
- Large datasets that are impractical to serialize/deserialize across steps
- Non-serializable objects (data connections, large models)
- Ephemeral shared space for data stores
- Hooking into existing PVCs that inject data into the pipeline
## Open questions
- How does workspace PVC coexist with artifact-based data passing? Per-step opt-in, global toggle, or automatic based on object type/size?
- How does this interact with Kale's marshalling system?
- What happens to pipeline portability when steps depend on a shared PVC?
- Can the workspace PVC be pre-populated (e.g. mounted datasets, model checkpoints)?
Contributor guide
Research direction
Start by reading the KEP when the architecture is available, then inspect Kale's pickle-based data passing and marshalling system and how KFP pipelines are generated. Done means workspace PVC passing works as an alternative to artifact-based passing, with the coexistence, portability, pre-population, and opt-in behavior questions resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- infrastructure, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100