The service is unavailable. Please try again shortly.

EC2 sometimes returns "503 service unavailable". We should treat this as temporary and try again later.

EC2Spot: check_instances_state: unhandled error: Traceback (most recent call last):
File "/srv/fuzzmanager.fuzzing.mozilla.org/FuzzManager/server/ec2spotmanager/CloudProvider/CloudProvider.py", line 54, in wrapper
return wrapped(*args, **kwds)
File "/srv/fuzzmanager.fuzzing.mozilla.org/FuzzManager/server/ec2spotmanager/CloudProvider/EC2SpotCloudProvider.py", line 171, in check_instances_state
boto_instances = cluster.find(filters={"tag-key": SPOTMGR_TAG + "-PoolId"})
File "/srv/fuzzmanager.fuzzing.mozilla.org/laniakea/laniakea/core/providers/ec2/manager.py", line 408, in find
reservations = self.retry_on_ec2_error(self.ec2.get_all_instances, instance_ids=instance_ids, filters=filters)
File "/srv/fuzzmanager.fuzzing.mozilla.org/laniakea/laniakea/core/providers/ec2/manager.py", line 50, in retry_on_ec2_error
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/boto/ec2/connection.py", line 585, in get_all_instances
max_results=max_results)
File "/usr/local/lib/python2.7/dist-packages/boto/ec2/connection.py", line 681, in get_all_reservations
[('item', Reservation)], verb='POST')
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1170, in get_list
response = self.make_request(action, params, path, verb)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1116, in make_request
return self._mexe(http_request)
File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1028, in _mexe
raise BotoServerError(response.status, response.reason, body)
BotoServerError: BotoServerError: 503 Service Unavailable
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>Unavailable</Code><Message>The service is unavailable. Please try again shortly.</Message></Error></Errors><RequestID>f6fabead-ac4d-43d0-a45d-fdda082d0c45</RequestID></Response>

Start with laniakea/core/providers/ec2/manager.py, especially retry_on_ec2_error, and trace its use from server/ec2spotmanager/CloudProvider/EC2SpotCloudProvider.py in check_instances_state. Confirm how BotoServerError responses are handled; done means an EC2 503 is treated as temporary and retried instead of surfacing as an unhandled error.