aws / aws/amazon-sagemaker-examples

Sagemaker Notebook Instances stuck at Stopping and Pending phases

Open
#1,094 7 comments 4 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

I am working with AWS sagemaker to automate model training in regulated and semi regulated environments. After making some changes today, I broke the instances (I needed to fix IAM roles, which I have since fixed). However, my notebooks instances go stuck at the Pending and Stopping phase respectively. I have waited a few hours and nothing has changed.

I have tried all of the commands below:

```bash
aws sagemaker start-notebook-instance --region us-east-1 --notebook-instance-name sagemaker-notebook-instance-187631372219
aws sagemaker stop-notebook-instance --region us-east-1 --notebook-instance-name sagemaker-notebook-instance-187631372219
aws sagemaker delete-notebook-instance --region us-east-1 --notebook-instance-name sagemaker-notebook-instance-187631372219
```

Unfortunately I am not able to get these to change state.

For Stopping State:
```
An error occurred (ValidationException) when calling the StartNotebookInstance operation: Status (Stopping) not in ([Stopped, Failed]). Unable to transition to (Pending) for Notebook Instance (arn:aws:sagemaker:us-east-1:187631372219:notebook-instance/sagemaker-notebook-instance-187631372219)
```

For Pending State:
```
An error occurred (ValidationException) when calling the StopNotebookInstance operation: Status (Pending) not in ([InService]). Unable to transition to (Stopping) for Notebook Instance (arn:aws:sagemaker:us-east-1:424869984157:notebook-instance/sagemaker-notebook-instance).
```
How can I stop and delete these instances so I can deploy the fixed versions?

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the AWS CLI start, stop, and delete commands and the reported SageMaker notebook statuses in the issue. Reproduce or inspect the Pending and Stopping transitions, then determine the supported way to recover or delete the affected instances so they can be redeployed.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, jupyter-notebook
Domain
cloud, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.