aws-samples / aws-samples/amazon-sagemaker-local-mode
question: Pyspark Processing Jobs in Local Mode?
Open
- Dominant language
- Python
- Stars
- 263
- Forks
- 69
- PR merge metrics
- No merged PRs in 30d
Description
Hello. I was wondering if there existed a tutorial, or current support for 1) running a pyspark processing job locally and 2) doing so with a custom base docker (EMR) image? I see a tutorial for Dask using a script processor, and also some code for an SKLearn based processor. My goal is to be able to basically set up a local testing/dev environment that uses sagemaker spark processor code. I'm guessing this is more complicated than the other use cases since this processor is usually backed by an EMR cluster.
Contributor guide
Assessment
This issue has not been assessed yet.