aws / aws/sagemaker-python-sdk

source_dir style automatic zipping in PySparkProcessor

Ouverte
#4,031 1 commentaire 0 réactions 1 personne assignée Réclamée par @mollyheamazon Voir sur GitHub
PySpark type: feature request
Langage dominant
Python
Étoiles
2.3k
Forks
1.3k
Merge moyen
1 j 22 h
PR mergées (30 j)
35

Description

**Describe the feature you'd like**

`PySparkProcessor` [currently supports](https://sagemaker.readthedocs.io/en/stable/amazon_sagemaker_processing.html#pysparkprocessor) creating jobs with additional dependency via multiple options `submit_py_files`, `submit_jar_files` and `submit_files`.

...But zipping up a folder of many Python scripts to pass to `submit_py_files` introduces extra complexity.

It would be better if we could use `source_dir`-like functionality (as per `FrameworkProcessor` and `Framework` estimators) where the SDK handles the compression process transparently.

**How would this feature be used? Please describe.**

When creating a PySparkProcessor with a local folder containing multiple Python scripts, the user could just specify the folder itself instead of having to prepare an archive of it.

This would reduce the likelihood of user error in archiving - the most common that I've seen being: Accidentally adding the whole scripts folder to the archive, so it extracts `my_cool_scripts/util.py` instead of the intended `./util.py`, which of course breaks module resolution in the job as cannot `import util`.

**Describe alternatives you've considered**

For backwards compatibility I guess we will have to keep all 3 `submit_*_files` options, although it would be nice if a `source_dir` option more consistent with Estimators was available? Simply extend the `submit_*_files` treatment to also accept local directories, in which case they get transparently archived and uploaded by the `PySparkProcessor`.

**Additional context**

Recently worked with a customer who was struggling with this and could not debug it themselves because all they saw was (`ImportError: Module util not found`), but the `submit_py_files` option seemed to be set up with their zip file.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.