aws / aws/amazon-sagemaker-examples
KernelRestarter: restarting kernel (1/5), keep random ports
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Describe the bug**
After I trained my model I try to make predictions using my y_test data and eventually the notebook restarts.:
Here are the errors shown on CloudWatch:
2022-07-04T15:53:56.528+02:00Copy[I 13:53:56.428 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports | [I 13:53:56.428 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
| 2022-07-04T15:53:56.529+02:00 | kernel e3a0e46b-c3c9-4ae1-b06c-fdf3f8e2d72d restarted
| 2022-07-04T15:53:56.529+02:00 | kernel e3a0e46b-c3c9-4ae1-b06c-fdf3f8e2d72d restarted
| 2022-07-04T15:53:56.529+02:00 | kernel e3a0e46b-c3c9-4ae1-b06c-fdf3f8e2d72d restarted
| 2022-07-04T15:53:56.529+02:00 | kernel e3a0e46b-c3c9-4ae1-b06c-fdf3f8e2d72d restarted
| 2022-07-04T15:53:56.529+02:00 | kernel e3a0e46b-c3c9-4ae1-b06c-fdf3f8e2d72d restarted
| 2022-07-04T15:53:56.529+02:00 | kernel e3a0e46b-c3c9-4ae1-b06c-fdf3f8e2d72d restarted
| 2022-07-04T15:54:01.095+02:00 | kernel e3a0e46b-c3c9-4ae1-b06c-fdf3f8e2d72d restarted
I am using instance type of ml.m5.12xlarge with a Volume Size of 10000GB EBS.
I also delete variables that I won't use anymore just in case and I still get my notebook restarted when trying to make predictions.
Contributor guide
Research direction
Start by reproducing the notebook restart during predictions with the reported ml.m5.12xlarge instance and inspect the CloudWatch kernel logs and resource usage around that step. Done means identifying a reproducible cause for the kernel restart and documenting a verified remediation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook, machine-learning
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100