geopython / geopython/GeoHealthCheck

Retry on http502 error during OWSlib request

Open
#386 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
92
Forks
75
PR merge metrics
No merged PRs in 30d

Description

_This issue is related to issue #366._

**Is your feature request related to a problem? Please describe.**
During a stress test of GHC (lots of resources with high frequency), I am getting mysterious http502 errors during `WebFeatureService` request in the `get_metadata` function inside `wfs.py`, `wms.py`, etc. These errors can be avoided by retrying requests.

**Describe the solution you'd like**
Currently, in [probe.py](https://github.com/geopython/GeoHealthCheck/blob/master/GeoHealthCheck/probe.py#L94) there is already something in place for this using the `create_requests_retry_session()` function in [util.py](https://github.com/geopython/GeoHealthCheck/blob/master/GeoHealthCheck/util.py#L267). The same principle could be applied to the request that is done in `get_metadata` functions of each probe (wfs.py, wms.py, etc.).
[wfs.py](https://github.com/geopython/GeoHealthCheck/blob/master/GeoHealthCheck/plugins/probe/wfs.py#L123)

This request however uses the `WebFeatureService()` function from OWSlib, so the best solution in my opinion would be to use the `create_requests_retry_session()` in OWSlib. Maybe other OWSlib users have also seen this http502 error? This fix might improve OWSlib for other users as well.

**Describe alternatives you've considered**
An alternative is a simple loop with an exception clause which retries the `WebFeatureService()` request up to 3 times:
```
def get_metadata(self, resource, version='1.1.0', retries=3):
"""
Get metadata, specific per Resource type.
:param resource:
:param version:
:return: Metadata object
request_headers = self.get_request_headers()
"""
for i in range(retries):
try:
return WebFeatureService(
resource.url,
version=version,
headers=request_headers)
except:
continue

raise Exception(f'Get Metadata Request Exceeded {retries} Retries')
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.