aws / aws/sagemaker-python-sdk
New sagemaker.processing.DataWranglerProcessor class
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 1.3k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 35
Description
**Describe the feature you'd like**
Being able to create a Processing Job starting from a SageMaker Data Wrangler Flow template. The Processor should use the Data Wrangler container image. As the template already includes input and output configurations, the DataWranglerProcessor class should map these values to Processing Inputs and Processing Outputs
**How would this feature be used? Please describe.**
For automated data preparation processes built using SageMaker Data Wrangler, I'd use this class to create a Processing Step in the context of a DAG, for example using SageMaker Pipelines, Step Functions which natively support Processing Jobs. And once a flow template is created, it would be easier to move across environments without having to export a Pyhton script from the Data Wrangler console in SageMaker Studio.
**Describe alternatives you've considered**
Currently, the only way to implement automation for a Data Wrangler Flow template is to manually export the process as a Jupyter notebook or Python script, then manually modify the generated script to adjust it to what is needed for the orchestration. Upon any changes in source files or data preparation steps, the output needs to be created again.
**Additional context**
Ideally, in the Processing Script, there should be a logic that reads the provided flow template, parses the inputs and outputs and maps them to corresponding ProcessingInputs and ProcessingOutputs before running the corresponding Processing Job.
Contributor guide
Assessment
This issue has not been assessed yet.