aws / aws/amazon-sagemaker-examples

Notebook exceeds memory?; OS Error : Cannot allocate memory

Open
#1,469 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

In the notebook: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/sagemaker_batch_transform/introduction_to_batch_transform/batch_transform_pca_dbscan_movie_clusters.ipynb

After training and within the "Batch Predictions PCA" I ran into a:
`OSError: [Errno 12] Cannot allocate memory`

How to recreate error:
1. Git clone: https://github.com/awslabs/amazon-sagemaker-examples.git
1. Use default kernel: conda_python3
2. "Run All"

What I tried:
Shut down & restarted notebook instance.

In the terminal I ran:
`free -h`

which returned:

```
total used free shared buffers cached
Mem: 3.9G 3.7G 202M 64K 97M 526M
```

It might be that the dataset copied from S3 is causing me to exceed memory?

Contributor guide

Open the contributing guide

Research direction

Start with sagemaker_batch_transform/introduction_to_batch_transform/batch_transform_pca_dbscan_movie_clusters.ipynb, using the conda_python3 kernel and the reported Run All steps. Reproduce the OSError during “Batch Predictions PCA” and investigate the notebook's memory use, including the dataset copied from S3. Done means the notebook completes that step without the allocation error or clearly documents the required resource or data change.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.