aws / aws/amazon-sagemaker-examples
Fit a Keras or Tensorflow Estimator on spark dataframe in pyspark using sagemaker
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
I have some questions:
1) I was trying to implement a "pyspark custom estimator" using this GitHub notebook:
https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker-spark/pyspark_mnist/pyspark_mnist_custom_estimator.ipynb
The first(primary) question is whether SageMakerEstimator would actually fit on Spark Dataframe input, if I pass a tensorflow docker image(hosting my Keras/Tensorflow Model) to it? More specifically, does SageMakerEstimator take care of converting the Spark DF to Numpy arrays to input to the tensorflow model? Or is there some additional coding steps that should exist in my docker image?
2) Could you point me to the code that was used to create the docker file that are listed here:
https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html
I want to write a similar code for Tensorflow or Keras. (Btw, I know how to build/push a docker file to ECR, my question is what code was used to have the XGB or k-means model?
3) Could you also point me to the code I can edit to implement "requestRowSerializer" and "responseRowDeserializer" of the SageMakerEstimator?
I need to know if there is any specific edits I need to make to have it work with a TF/Keras docker image
I would appreciate if you could let me know if I am on the write path to Fit a Keras or Tensorflow Estimator on spark dataframe in pyspark using sagemaker.
Thanks,
Contributor guide
Assessment
This issue has not been assessed yet.