kubeflow / kubeflow/sdk

Share embedded function execution script template between Trainer and Spark

Open
#638 9 comments 0 reactions 1 assignee Claimed by @pritesh0089 View on GitHub
area/spark
Dominant language
Python
Stars
148
Forks
262
Avg merge
1d 2h
Merged PRs (30d)
1

Description

## Summary

Trainer and Spark currently maintain separate templates for embedding Python function source code into the container at runtime.

The Spark implementation uses a quoted heredoc delimiter to avoid shell expansion and command substitution, while Trainer uses a similar template with a plain `EOM` delimiter. The overall execution flow is conceptually the same, but the implementations have diverged.

## Proposal

Move the embedded function execution script template into a common module so both Trainer and Spark reuse the same implementation.

The shared helper should:
- Generate the embedded Python execution script.
- Be reusable by both Trainer and Spark.
- Preserve the robustness of the quoted heredoc delimiter (or another agreed-upon implementation).
- Reduce duplicated logic across the two clients.

## Benefits

- Consistent behavior between Trainer and Spark.
- Less duplicated code.
- Easier maintenance for future improvements.

ref- https://github.com/kubeflow/sdk/pull/629#discussion_r3634877066

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.