aws / aws/amazon-sagemaker-examples

Failed to created shared resource stack

Open
#1,673 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
11k
Forks
7k
Avg merge
8h 29m
Merged PRs (30d)
8

Description

In `bandits_statlog_vw_customEnv.ipynb`, the following line
`bandits_experiment = ExperimentManager(config, experiment_id=experiment_name)`
gives the following error

```
WaiterError Traceback (most recent call last)
~/SageMaker/amazon-sagemaker-examples/reinforcement_learning/bandits_statlog_vw_customEnv/common/sagemaker_rl/orchestrator/resource_manager.py in _wait_for_cf_stack_create_to_complete(self)
221 'Delay': 10,
--> 222 'MaxAttempts': 60
223 }

~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/waiter.py in wait(self, **kwargs)
52 def wait(self, **kwargs):
---> 53 Waiter.wait(self, **kwargs)
54

~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/waiter.py in wait(self, **kwargs)
322 reason='Waiter encountered a terminal failure state',
--> 323 last_response=response,
324 )

WaiterError: Waiter StackCreateComplete failed: Waiter encountered a terminal failure state

During handling of the above exception, another exception occurred:

Exception Traceback (most recent call last)
in
3 # So, make experimend_id as short as possible
4 experiment_name = "bandits-exp-1"
----> 5 bandits_experiment = ExperimentManager(config, experiment_id=experiment_name)

~/SageMaker/amazon-sagemaker-examples/reinforcement_learning/bandits_statlog_vw_customEnv/common/sagemaker_rl/orchestrator/workflow/manager/experiment_manager.py in __init__(self, config, experiment_id, training_workflow_metadata, hosting_workflow_metadata, joining_workflow_metadata, evaluation_workflow_metadata)
722 self.resource_manager = ResourceManager(self.config.get("resource", {}),
723 boto_session=self.boto_session)
--> 724 self.resource_manager.create_shared_resource_if_not_exist()
725
726 # init clients

~/SageMaker/amazon-sagemaker-examples/reinforcement_learning/bandits_statlog_vw_customEnv/common/sagemaker_rl/orchestrator/resource_manager.py in create_shared_resource_if_not_exist(self)
78 logger.info("Creating a new CloudFormation stack for Shared Resources. " \
79 "You can always reuse this StackName in your other experiments")
---> 80 self._create_new_cloudformation_stack()
81
82 # use Output Resources Names from CloudFromation stack

~/SageMaker/amazon-sagemaker-examples/reinforcement_learning/bandits_statlog_vw_customEnv/common/sagemaker_rl/orchestrator/resource_manager.py in _create_new_cloudformation_stack(self)
198 self._wait_for_cf_stack_create_to_complete()
199 return False
--> 200 raise(e)
201
202 def _get_cf_stack_events_link(self):

~/SageMaker/amazon-sagemaker-examples/reinforcement_learning/bandits_statlog_vw_customEnv/common/sagemaker_rl/orchestrator/resource_manager.py in _create_new_cloudformation_stack(self)
186 )
187 logger.info("Creating CloudFormation Stack for shared resource!")
--> 188 self._wait_for_cf_stack_create_to_complete()
189 return True
190 except Exception as e:

~/SageMaker/amazon-sagemaker-examples/reinforcement_learning/bandits_statlog_vw_customEnv/common/sagemaker_rl/orchestrator/resource_manager.py in _wait_for_cf_stack_create_to_complete(self)
226 logger.error(e)
227 logger.error("Failed to Create Stack with name {} ".format(self.shared_resource_stack_name))
--> 228 raise Exception(f"Failed to Create Shared Resource Stack. "
229 f"Please debug the stack here: {self._get_cf_stack_events_link()}"
230 )

Exception: Failed to Create Shared Resource Stack. Please debug the stack here: https://us-west-1.console.aws.amazon.com/cloudformation/home?region=us-west-1#/stacks/events?stackId=BanditsSharedResourceStack
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.