aws / aws/amazon-sagemaker-examples

[Bug Report] Bug in the Cloud Formation Configuration File for Distributed Tensorflow Mask RCNN

Open
#2,072 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

**Link to the notebook**
https://github.com/aws/amazon-sagemaker-examples/tree/master/advanced_functionality/distributed_tensorflow_mask_rcnn#create-sagemaker-notebook-instance-in-a-new-vpc

**Describe the bug**
Running `stack-sm.sh` as described results in the following error during Stack creation:

> Notebook Instance Lifecycle Config 'arn:aws:sagemaker:us-west-2:961437118368:notebook-instance-lifecycle-config/notebookstartconfig-qpgi0kirjpq2' for Notebook Instance 'arn:aws:sagemaker:us-west-2:961437118368:notebook-instance/dv-sagemaker-nb-1615419486' took longer than 5 minutes. Please check your CloudWatch logs for more details if your Notebook Instance has Internet access.

Digging deeper into Cloud Watch, under `/aws/sagemaker/NotebookInstances`, we can find the culprit:

> /tmp/OnStart_2021-03-10-23-084omt5f8u: line 2: set: -d: invalid option
> set: usage: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]

This means that the error is happening somewhere during the aws cli call in this [line](https://github.com/aws/amazon-sagemaker-examples/blob/master/advanced_functionality/distributed_tensorflow_mask_rcnn/stack-sm.sh#L46), which consumes `cfn-sm.yaml`. So the error is somewhere [here](https://github.com/aws/amazon-sagemaker-examples/blob/master/advanced_functionality/distributed_tensorflow_mask_rcnn/cfn-sm.yaml#L364-L372)

**To Reproduce**
Just run `stack-sm.sh` as described in the _Create SageMaker notebook instance in a VPC_ of the README and look at Cloud Watch under `/aws/sagemaker/NotebookInstances`.

**Possible Solution**
I changed lines 364 - 372 to the following in order to localize the error. But this made the error disappear. I think there is a `|+` missing somewhere:

```
- ''
- - '#!/bin/bash'
- |+

- echo test before
- |+

- set -e
- |+

- echo test after
- |+

- mkdir -p
- ' '
```

Contributor guide

Open the contributing guide

Research direction

Start with advanced_functionality/distributed_tensorflow_mask_rcnn/stack-sm.sh at line 46 and cfn-sm.yaml at lines 364-372. Run stack-sm.sh as described in the README, then inspect CloudWatch under /aws/sagemaker/NotebookInstances to reproduce the lifecycle-config failure. Done means the stack creates the notebook instance without the five-minute timeout or invalid set option error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, shell
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.