aws-samples / aws-samples/amazon-sagemaker-local-mode

question: Pyspark Processing Jobs in Local Mode?

Open
#19 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
263
Forks
69
PR merge metrics
No merged PRs in 30d

Description

Hello. I was wondering if there existed a tutorial, or current support for 1) running a pyspark processing job locally and 2) doing so with a custom base docker (EMR) image? I see a tutorial for Dask using a script processor, and also some code for an SKLearn based processor. My goal is to be able to basically set up a local testing/dev environment that uses sagemaker spark processor code. I'm guessing this is more complicated than the other use cases since this processor is usually backed by an EMR cluster.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.