aws / aws/sagemaker-python-sdk

source_dir style automatic zipping in PySparkProcessor

Open
#4,031 1 comment 0 reactions 1 assignee Claimed by @mollyheamazon View on GitHub
PySpark type: feature request
Dominant language
Python
Stars
2.3k
Forks
1.3k
Avg merge
1d 22h
Merged PRs (30d)
35

Description

**Describe the feature you'd like**

`PySparkProcessor` [currently supports](https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_processing.html#pysparkprocessor) creating jobs with additional dependency via multiple options `submit_py_files`, `submit_jar_files` and `submit_files`.

...But zipping up a folder of many Python scripts to pass to `submit_py_files` introduces extra complexity.

It would be better if we could use `source_dir`-like functionality (as per `FrameworkProcessor` and `Framework` estimators) where the SDK handles the compression process transparently.

**How would this feature be used? Please describe.**

When creating a PySparkProcessor with a local folder containing multiple Python scripts, the user could just specify the folder itself instead of having to prepare an archive of it.

This would reduce the likelihood of user error in archiving - the most common that I've seen being: Accidentally adding the whole scripts folder to the archive, so it extracts `my_cool_scripts/util.py` instead of the intended `./util.py`, which of course breaks module resolution in the job as cannot `import util`.

**Describe alternatives you've considered**

For backwards compatibility I guess we will have to keep all 3 `submit_*_files` options, although it would be nice if a `source_dir` option more consistent with Estimators was available? Simply extend the `submit_*_files` treatment to also accept local directories, in which case they get transparently archived and uploaded by the `PySparkProcessor`.

**Additional context**

Recently worked with a customer who was struggling with this and could not debug it themselves because all they saw was (`ImportError: Module util not found`), but the `submit_py_files` option seemed to be set up with their zip file.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.