Share embedded function execution script template between Trainer and Spark
- Dominant language
- Python
- Stars
- 148
- Forks
- 262
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 1
Description
## Summary
Trainer and Spark currently maintain separate templates for embedding Python function source code into the container at runtime.
The Spark implementation uses a quoted heredoc delimiter to avoid shell expansion and command substitution, while Trainer uses a similar template with a plain `EOM` delimiter. The overall execution flow is conceptually the same, but the implementations have diverged.
## Proposal
Move the embedded function execution script template into a common module so both Trainer and Spark reuse the same implementation.
The shared helper should:
- Generate the embedded Python execution script.
- Be reusable by both Trainer and Spark.
- Preserve the robustness of the quoted heredoc delimiter (or another agreed-upon implementation).
- Reduce duplicated logic across the two clients.
## Benefits
- Consistent behavior between Trainer and Spark.
- Less duplicated code.
- Easier maintenance for future improvements.
ref- https://github.com/kubeflow/sdk/pull/629#discussion_r3634877066
Contributor guide
Assessment
This issue has not been assessed yet.