aws / aws/amazon-sagemaker-examples
Notebook exceeds memory?; OS Error : Cannot allocate memory
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
In the notebook: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker_batch_transform/introduction_to_batch_transform/batch_transform_pca_dbscan_movie_clusters.ipynb
After training and within the "Batch Predictions PCA" I ran into a:
`OSError: [Errno 12] Cannot allocate memory`
How to recreate error:
1. Git clone: https://github.com/awslabs/amazon-sagemaker-examples.git
1. Use default kernel: conda_python3
2. "Run All"
What I tried:
Shut down & restarted notebook instance.
In the terminal I ran:
`free -h`
which returned:
```
total used free shared buffers cached
Mem: 3.9G 3.7G 202M 64K 97M 526M
```
It might be that the dataset copied from S3 is causing me to exceed memory?
Contributor guide
Research direction
Start with sagemaker_batch_transform/introduction_to_batch_transform/batch_transform_pca_dbscan_movie_clusters.ipynb, using the conda_python3 kernel and the reported Run All steps. Reproduce the OSError during “Batch Predictions PCA” and investigate the notebook's memory use, including the dataset copied from S3. Done means the notebook completes that step without the allocation error or clearly documents the required resource or data change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100