aws / aws/amazon-sagemaker-examples

KernelRestarter: restarting kernel (1/5), keep random ports

Open
#3,493 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Describe the bug**
After I trained my model I try to make predictions using my y_test data and eventually the notebook restarts.:

Here are the errors shown on CloudWatch:

2022-07-04T15:53:56.528+02:00Copy[I 13:53:56.428 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports | [I 13:53:56.428 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
  | 2022-07-04T15:53:56.529+02:00 | kernel e3a0e46b-c3c9-4ae1-b06c-fdf3f8e2d72d restarted
  | 2022-07-04T15:53:56.529+02:00 | kernel e3a0e46b-c3c9-4ae1-b06c-fdf3f8e2d72d restarted
  | 2022-07-04T15:53:56.529+02:00 | kernel e3a0e46b-c3c9-4ae1-b06c-fdf3f8e2d72d restarted
  | 2022-07-04T15:53:56.529+02:00 | kernel e3a0e46b-c3c9-4ae1-b06c-fdf3f8e2d72d restarted
  | 2022-07-04T15:53:56.529+02:00 | kernel e3a0e46b-c3c9-4ae1-b06c-fdf3f8e2d72d restarted
  | 2022-07-04T15:53:56.529+02:00 | kernel e3a0e46b-c3c9-4ae1-b06c-fdf3f8e2d72d restarted
  | 2022-07-04T15:54:01.095+02:00 | kernel e3a0e46b-c3c9-4ae1-b06c-fdf3f8e2d72d restarted

I am using instance type of ml.m5.12xlarge with a Volume Size of 10000GB EBS.

I also delete variables that I won't use anymore just in case and I still get my notebook restarted when trying to make predictions.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the notebook restart during predictions with the reported ml.m5.12xlarge instance and inspect the CloudWatch kernel logs and resource usage around that step. Done means identifying a reproducible cause for the kernel restart and documenting a verified remediation.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook, machine-learning
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.