aws / aws/amazon-sagemaker-examples

04_Ingest_data_with_EMR.ipynb [Bug Report]

Open
#3,122 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
https://sagemaker-examples.readthedocs.io/en/latest/ingest_data/04_Ingest_data_with_EMR.html

**Describe the bug**
After I link Sagemaker notebook instance to EMR I run notebook and get error.
No Module Found sklearn

**To reproduce**
Run notebook after linking Sagemaker to EMR
sklearn is missing in "local" (pysprk kernel) and hence we cannot use it.
I was able to find workaround by running all commands prior to spark commands using regular python notebook (sklearn is available in that kernel but not in pyspark kernel).
Then I executed rest of the commands in pyspark kernel.

This notebook should remove dependency on sklearn. We are just using dataset from there and putting them into files on S3.
Those files could be made readily available on S3.

**Logs**
If applicable, add logs to help explain your problem.
You may also attach an `.ipynb` file to this issue if it includes relevant logs or output.

Contributor guide

Open the contributing guide

Research direction

Open 04_Ingest_data_with_EMR.ipynb from the linked notebook page and run it after linking a SageMaker notebook instance to EMR, focusing on the pyspark kernel where sklearn is missing. Remove the notebook's dependency on sklearn and verify that the data-ingestion steps run successfully in the pyspark kernel without the workaround of switching kernels.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, python, scikit-learn
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.