kubeflow / kubeflow/kale

feature(backend): Support KFP Pipeline Workspace for inter-step data passing via shared PVC

Open
#856 1 comment 0 reactions 0 assignees View on GitHub
2.x candidate area/backend
Dominant language
Python
Stars
703
Forks
161
Avg merge
6d 3h
Merged PRs (30d)
10

Description

## Summary

Kale currently passes data between pipeline steps using pickle-based serialization. This works for small objects but breaks down for large datasets, huge models, non-serializable Python objects (e.g. database connections), and ephemeral data stores.

KFP supports a workspace PVC model where steps read/write files via a shared filesystem, matching the notebook's mental model more closely.

A KEP will detail the architecture.

## Proposal

Support KFP's native workspace PVC as an alternative to pickle-based data passing between steps. Steps would share a filesystem path, enabling:

- Large datasets that are impractical to serialize/deserialize across steps
- Non-serializable objects (data connections, large models)
- Ephemeral shared space for data stores
- Hooking into existing PVCs that inject data into the pipeline

## Open questions

- How does workspace PVC coexist with artifact-based data passing? Per-step opt-in, global toggle, or automatic based on object type/size?
- How does this interact with Kale's marshalling system?
- What happens to pipeline portability when steps depend on a shared PVC?
- Can the workspace PVC be pre-populated (e.g. mounted datasets, model checkpoints)?

Contributor guide

Open the contributing guide

Research direction

Start by reading the KEP when the architecture is available, then inspect Kale's pickle-based data passing and marshalling system and how KFP pipelines are generated. Done means workspace PVC passing works as an alternative to artifact-based passing, with the coexistence, portability, pre-population, and opt-in behavior questions resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
infrastructure, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.