aws / aws/amazon-sagemaker-examples

Fit a Keras or Tensorflow Estimator on spark dataframe in pyspark using sagemaker

Open
#658 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

I have some questions:
1) I was trying to implement a "pyspark custom estimator" using this GitHub notebook:
https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-spark/pyspark_mnist/pyspark_mnist_custom_estimator.ipynb
The first(primary) question is whether SageMakerEstimator would actually fit on Spark Dataframe input, if I pass a tensorflow docker image(hosting my Keras/Tensorflow Model) to it? More specifically, does SageMakerEstimator take care of converting the Spark DF to Numpy arrays to input to the tensorflow model? Or is there some additional coding steps that should exist in my docker image?

2) Could you point me to the code that was used to create the docker file that are listed here:
https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html
I want to write a similar code for Tensorflow or Keras. (Btw, I know how to build/push a docker file to ECR, my question is what code was used to have the XGB or k-means model?

3) Could you also point me to the code I can edit to implement "requestRowSerializer" and "responseRowDeserializer" of the SageMakerEstimator?
I need to know if there is any specific edits I need to make to have it work with a TF/Keras docker image

I would appreciate if you could let me know if I am on the write path to Fit a Keras or Tensorflow Estimator on spark dataframe in pyspark using sagemaker.

Thanks,

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.