aws / aws/amazon-sagemaker-examples
Sagemaker Notebook Instances stuck at Stopping and Pending phases
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
I am working with AWS sagemaker to automate model training in regulated and semi regulated environments. After making some changes today, I broke the instances (I needed to fix IAM roles, which I have since fixed). However, my notebooks instances go stuck at the Pending and Stopping phase respectively. I have waited a few hours and nothing has changed.
I have tried all of the commands below:
```bash
aws sagemaker start-notebook-instance --region us-east-1 --notebook-instance-name sagemaker-notebook-instance-187631372219
aws sagemaker stop-notebook-instance --region us-east-1 --notebook-instance-name sagemaker-notebook-instance-187631372219
aws sagemaker delete-notebook-instance --region us-east-1 --notebook-instance-name sagemaker-notebook-instance-187631372219
```
Unfortunately I am not able to get these to change state.
For Stopping State:
```
An error occurred (ValidationException) when calling the StartNotebookInstance operation: Status (Stopping) not in ([Stopped, Failed]). Unable to transition to (Pending) for Notebook Instance (arn:aws:sagemaker:us-east-1:187631372219:notebook-instance/sagemaker-notebook-instance-187631372219)
```
For Pending State:
```
An error occurred (ValidationException) when calling the StopNotebookInstance operation: Status (Pending) not in ([InService]). Unable to transition to (Stopping) for Notebook Instance (arn:aws:sagemaker:us-east-1:424869984157:notebook-instance/sagemaker-notebook-instance).
```
How can I stop and delete these instances so I can deploy the fixed versions?
Contributor guide
Research direction
Start by reviewing the AWS CLI start, stop, and delete commands and the reported SageMaker notebook statuses in the issue. Reproduce or inspect the Pending and Stopping transitions, then determine the supported way to recover or delete the affected instances so they can be redeployed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, jupyter-notebook
- Domain
- cloud, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100