aws / aws/amazon-sagemaker-examples
04_Ingest_data_with_EMR.ipynb [Bug Report]
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
https://sagemaker-examples.readthedocs.io/en/latest/ingest_data/04_Ingest_data_with_EMR.html
**Describe the bug**
After I link Sagemaker notebook instance to EMR I run notebook and get error.
No Module Found sklearn
**To reproduce**
Run notebook after linking Sagemaker to EMR
sklearn is missing in "local" (pysprk kernel) and hence we cannot use it.
I was able to find workaround by running all commands prior to spark commands using regular python notebook (sklearn is available in that kernel but not in pyspark kernel).
Then I executed rest of the commands in pyspark kernel.
This notebook should remove dependency on sklearn. We are just using dataset from there and putting them into files on S3.
Those files could be made readily available on S3.
**Logs**
If applicable, add logs to help explain your problem.
You may also attach an `.ipynb` file to this issue if it includes relevant logs or output.
Contributor guide
Research direction
Open 04_Ingest_data_with_EMR.ipynb from the linked notebook page and run it after linking a SageMaker notebook instance to EMR, focusing on the pyspark kernel where sklearn is missing. Remove the notebook's dependency on sklearn and verify that the data-ingestion steps run successfully in the pyspark kernel without the workaround of switching kernels.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, python, scikit-learn
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100