aws / aws/amazon-sagemaker-examples
Failed to created shared resource stack
- Dominant language
- Jupyter Notebook
- Stars
- 11k
- Forks
- 7k
- Avg merge
- 8h 29m
- Merged PRs (30d)
- 8
Description
In `bandits_statlog_vw_customEnv.ipynb`, the following line
`bandits_experiment = ExperimentManager(config, experiment_id=experiment_name)`
gives the following error
```
WaiterError Traceback (most recent call last)
~/SageMaker/amazon-sagemaker-examples/reinforcement_learning/bandits_statlog_vw_customEnv/common/sagemaker_rl/orchestrator/resource_manager.py in _wait_for_cf_stack_create_to_complete(self)
221 'Delay': 10,
--> 222 'MaxAttempts': 60
223 }
~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/waiter.py in wait(self, **kwargs)
52 def wait(self, **kwargs):
---> 53 Waiter.wait(self, **kwargs)
54
~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/waiter.py in wait(self, **kwargs)
322 reason='Waiter encountered a terminal failure state',
--> 323 last_response=response,
324 )
WaiterError: Waiter StackCreateComplete failed: Waiter encountered a terminal failure state
During handling of the above exception, another exception occurred:
Exception Traceback (most recent call last)
in
3 # So, make experimend_id as short as possible
4 experiment_name = "bandits-exp-1"
----> 5 bandits_experiment = ExperimentManager(config, experiment_id=experiment_name)
~/SageMaker/amazon-sagemaker-examples/reinforcement_learning/bandits_statlog_vw_customEnv/common/sagemaker_rl/orchestrator/workflow/manager/experiment_manager.py in __init__(self, config, experiment_id, training_workflow_metadata, hosting_workflow_metadata, joining_workflow_metadata, evaluation_workflow_metadata)
722 self.resource_manager = ResourceManager(self.config.get("resource", {}),
723 boto_session=self.boto_session)
--> 724 self.resource_manager.create_shared_resource_if_not_exist()
725
726 # init clients
~/SageMaker/amazon-sagemaker-examples/reinforcement_learning/bandits_statlog_vw_customEnv/common/sagemaker_rl/orchestrator/resource_manager.py in create_shared_resource_if_not_exist(self)
78 logger.info("Creating a new CloudFormation stack for Shared Resources. " \
79 "You can always reuse this StackName in your other experiments")
---> 80 self._create_new_cloudformation_stack()
81
82 # use Output Resources Names from CloudFromation stack
~/SageMaker/amazon-sagemaker-examples/reinforcement_learning/bandits_statlog_vw_customEnv/common/sagemaker_rl/orchestrator/resource_manager.py in _create_new_cloudformation_stack(self)
198 self._wait_for_cf_stack_create_to_complete()
199 return False
--> 200 raise(e)
201
202 def _get_cf_stack_events_link(self):
~/SageMaker/amazon-sagemaker-examples/reinforcement_learning/bandits_statlog_vw_customEnv/common/sagemaker_rl/orchestrator/resource_manager.py in _create_new_cloudformation_stack(self)
186 )
187 logger.info("Creating CloudFormation Stack for shared resource!")
--> 188 self._wait_for_cf_stack_create_to_complete()
189 return True
190 except Exception as e:
~/SageMaker/amazon-sagemaker-examples/reinforcement_learning/bandits_statlog_vw_customEnv/common/sagemaker_rl/orchestrator/resource_manager.py in _wait_for_cf_stack_create_to_complete(self)
226 logger.error(e)
227 logger.error("Failed to Create Stack with name {} ".format(self.shared_resource_stack_name))
--> 228 raise Exception(f"Failed to Create Shared Resource Stack. "
229 f"Please debug the stack here: {self._get_cf_stack_events_link()}"
230 )
Exception: Failed to Create Shared Resource Stack. Please debug the stack here: https://us-west-1.console.aws.amazon.com/cloudformation/home?region=us-west-1#/stacks/events?stackId=BanditsSharedResourceStack
```
Contributor guide
Assessment
This issue has not been assessed yet.