aws / aws/sagemaker-python-sdk
source_dir style automatic zipping in PySparkProcessor
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 1.3k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 35
Description
**Describe the feature you'd like**
`PySparkProcessor` [currently supports](https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_processing.html#pysparkprocessor) creating jobs with additional dependency via multiple options `submit_py_files`, `submit_jar_files` and `submit_files`.
...But zipping up a folder of many Python scripts to pass to `submit_py_files` introduces extra complexity.
It would be better if we could use `source_dir`-like functionality (as per `FrameworkProcessor` and `Framework` estimators) where the SDK handles the compression process transparently.
**How would this feature be used? Please describe.**
When creating a PySparkProcessor with a local folder containing multiple Python scripts, the user could just specify the folder itself instead of having to prepare an archive of it.
This would reduce the likelihood of user error in archiving - the most common that I've seen being: Accidentally adding the whole scripts folder to the archive, so it extracts `my_cool_scripts/util.py` instead of the intended `./util.py`, which of course breaks module resolution in the job as cannot `import util`.
**Describe alternatives you've considered**
For backwards compatibility I guess we will have to keep all 3 `submit_*_files` options, although it would be nice if a `source_dir` option more consistent with Estimators was available? Simply extend the `submit_*_files` treatment to also accept local directories, in which case they get transparently archived and uploaded by the `PySparkProcessor`.
**Additional context**
Recently worked with a customer who was struggling with this and could not debug it themselves because all they saw was (`ImportError: Module util not found`), but the `submit_py_files` option seemed to be set up with their zip file.
Contributor guide
Assessment
This issue has not been assessed yet.