aws / aws/amazon-sagemaker-examples
[Bug Report] Bug in the Cloud Formation Configuration File for Distributed Tensorflow Mask RCNN
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/tree/master/advanced_functionality/distributed_tensorflow_mask_rcnn#create-sagemaker-notebook-instance-in-a-new-vpc
**Describe the bug**
Running `stack-sm.sh` as described results in the following error during Stack creation:
> Notebook Instance Lifecycle Config 'arn:aws:sagemaker:us-west-2:961437118368:notebook-instance-lifecycle-config/notebookstartconfig-qpgi0kirjpq2' for Notebook Instance 'arn:aws:sagemaker:us-west-2:961437118368:notebook-instance/dv-sagemaker-nb-1615419486' took longer than 5 minutes. Please check your CloudWatch logs for more details if your Notebook Instance has Internet access.
Digging deeper into Cloud Watch, under `/aws/sagemaker/NotebookInstances`, we can find the culprit:
> /tmp/OnStart_2021-03-10-23-084omt5f8u: line 2: set: -d: invalid option
> set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
This means that the error is happening somewhere during the aws cli call in this [line](https://github.com/aws/amazon-sagemaker-examples/blob/master/advanced_functionality/distributed_tensorflow_mask_rcnn/stack-sm.sh#L46), which consumes `cfn-sm.yaml`. So the error is somewhere [here](https://github.com/aws/amazon-sagemaker-examples/blob/master/advanced_functionality/distributed_tensorflow_mask_rcnn/cfn-sm.yaml#L364-L372)
**To Reproduce**
Just run `stack-sm.sh` as described in the _Create SageMaker notebook instance in a VPC_ of the README and look at Cloud Watch under `/aws/sagemaker/NotebookInstances`.
**Possible Solution**
I changed lines 364 - 372 to the following in order to localize the error. But this made the error disappear. I think there is a `|+` missing somewhere:
```
- ''
- - '#!/bin/bash'
- |+
- echo test before
- |+
- set -e
- |+
- echo test after
- |+
- mkdir -p
- ' '
```
Contributor guide
Research direction
Start with advanced_functionality/distributed_tensorflow_mask_rcnn/stack-sm.sh at line 46 and cfn-sm.yaml at lines 364-372. Run stack-sm.sh as described in the README, then inspect CloudWatch under /aws/sagemaker/NotebookInstances to reproduce the lifecycle-config failure. Done means the stack creates the notebook instance without the five-minute timeout or invalid set option error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, shell
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100