kubeflow / kubeflow/sdk

feat(trainer): extend local execution backends with customizable runtime options

Open
#624 1 comment 0 reactions 0 assignees View on GitHub
kind/feature needs-triage
Dominant language
Python
Stars
148
Forks
262
Avg merge
1d 2h
Merged PRs (30d)
1

Description

### What you would like to be added?

Currently, the local execution backends (`ContainerBackend` and `LocalProcessBackend`) provide baseline support for running training jobs locally without requiring a full Kubernetes cluster. However, advanced local customization is limited—for example, users cannot easily mount local volumes/directories, inject custom environment variables, or specify CPU/memory resource limits when running containers or sub-processes locally.

I propose extending the configuration capabilities of `ContainerBackendConfig` and `LocalProcessBackendConfig` to support additional developer-friendly options:

1. **`ContainerBackendConfig` Enhancements:**
* Support for volume mounts (`volumes` / `volume_mounts`) to link local datasets directly into the execution container.
* Configurable environment variables (`env`) passed into the container runtime.
* Basic resource constraints (e.g., `--cpus`, `--gpus`, `--memory`).

2. **`LocalProcessBackendConfig` Enhancements:**
* Support for passing custom environment variables to subprocesses.
* Improved error capturing and cleanup on process termination/interruption.

### Why is this needed?

Spinning up a full KinD or Minikube cluster adds significant overhead and setup time for simple local debugging sessions. Additionally, running containers manually outside the SDK breaks the unified SDK workflow. Extending local backends improves the developer loop and makes testing pipelines locally much faster.

I would like to work on this issue!

### Love this feature?

Give it a 👍 We prioritize the features with most 👍

Contributor guide

Open the contributing guide

Research direction

Start with the definitions and current behavior of ContainerBackendConfig and LocalProcessBackendConfig, then trace how ContainerBackend and LocalProcessBackend launch local jobs. The work is done when volume mounts, environment variables, and resource constraints are supported for containers, while subprocess environment handling, error capture, and cleanup on interruption are addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.