aws / aws/sagemaker-python-sdk

source_dir style automatic zipping in PySparkProcessor

オープン
#4,031 コメント 1 件 リアクション 0 件 担当者 1 名 @mollyheamazon が担当を希望しています GitHub で見る
PySpark type: feature request
主要言語
Python
スター
2.3k
フォーク
1.3k
平均マージ
1日 22時間
マージ済み PR(30日)
35

説明

**Describe the feature you'd like**

`PySparkProcessor` [currently supports](https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_processing.html#pysparkprocessor) creating jobs with additional dependency via multiple options `submit_py_files`, `submit_jar_files` and `submit_files`.

...But zipping up a folder of many Python scripts to pass to `submit_py_files` introduces extra complexity.

It would be better if we could use `source_dir`-like functionality (as per `FrameworkProcessor` and `Framework` estimators) where the SDK handles the compression process transparently.

**How would this feature be used? Please describe.**

When creating a PySparkProcessor with a local folder containing multiple Python scripts, the user could just specify the folder itself instead of having to prepare an archive of it.

This would reduce the likelihood of user error in archiving - the most common that I've seen being: Accidentally adding the whole scripts folder to the archive, so it extracts `my_cool_scripts/util.py` instead of the intended `./util.py`, which of course breaks module resolution in the job as cannot `import util`.

**Describe alternatives you've considered**

For backwards compatibility I guess we will have to keep all 3 `submit_*_files` options, although it would be nice if a `source_dir` option more consistent with Estimators was available? Simply extend the `submit_*_files` treatment to also accept local directories, in which case they get transparently archived and uploaded by the `PySparkProcessor`.

**Additional context**

Recently worked with a customer who was struggling with this and could not debug it themselves because all they saw was (`ImportError: Module util not found`), but the `submit_py_files` option seemed to be set up with their zip file.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。