cloudtools / cloudtools/stacker

Rate exceed when building a lot of stacks in paralle

Open
#337 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
707
Forks
162
PR merge metrics
No merged PRs in 30d

Description

When we build a sequence of stacks where many can be built in parallel, e start hitting the rate limit for querying the AWS API:
```
Process Process-9:
Traceback (most recent call last):
File "/opt/rh/python27/root/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/opt/rh/python27/root/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/var/lib/jenkins/jobs/mongodb-opsmanager/workspace/env/lib/python2.7/site-packages/stacker/providers/aws/default.py", line 152, in tail_stack
self.tail_stack(stack, retries=retries + 1, **kwargs)
File "/var/lib/jenkins/jobs/mongodb-opsmanager/workspace/env/lib/python2.7/site-packages/stacker/providers/aws/default.py", line 146, in tail_stack
include_initial=False)
File "/var/lib/jenkins/jobs/mongodb-opsmanager/workspace/env/lib/python2.7/site-packages/stacker/providers/aws/default.py", line 196, in tail
events = self.get_events(stack_name)
File "/var/lib/jenkins/jobs/mongodb-opsmanager/workspace/env/lib/python2.7/site-packages/stacker/providers/aws/default.py", line 173, in get_events
StackName=stackname
File "/var/lib/jenkins/jobs/mongodb-opsmanager/workspace/env/lib/python2.7/site-packages/botocore/client.py", line 253, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/lib/jenkins/jobs/mongodb-opsmanager/workspace/env/lib/python2.7/site-packages/botocore/client.py", line 543, in _make_api_call
raise error_class(parsed_response, operation_name)
ClientError: An error occurred (Throttling) when calling the DescribeStackEvents operation (reached max retries: 4): Rate exceeded
```

This is with 1.0.0.

Seems like this should be handled more gracefully by stacker, perhaps.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.