Imageomics / Imageomics/cautious-robot

Update retry algorithm to be more robust

Open
#6 6 comments 0 reactions 1 assignee Claimed by @NetZissou View on GitHub
enhancement structure
Dominant language
Python
Stars
5
Forks
1
PR merge metrics
No merged PRs in 30d

Description

> That being said, retry algorithms (at least robust ones) for internet protocols are normally written with an exponential escalation of wait time (such as 1, 2, 4, 8, 16, 32, .. seconds). In that case, a user may want to specify at which point to give up and log a failure, for example `--max-retries` and/or `--max-wait`.

_Originally posted by @hlapp in https://github.com/Imageomics/cautious-robot/pull/1#discussion_r1626575092_

> reply: It's waiting after a failed attempt if the response is any of the following: 429, 500, 502, 503, 504. It doesn't have a wait between successful downloads. It has a max number of times to retry on the designated responses, but otherwise just logs the response in the error log (along with the index, filename, and url).

> Setting a maximum wait time on a request would probably be a good idea as well. [urllib3.request](https://urllib3.readthedocs.io/en/latest/reference/urllib3.request.html) seems to handle much of this when also passed a [Retry object](https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.Retry). @thompsonmj had also suggested [HTTPAdapter](https://requests.readthedocs.io/en/latest/user/advanced/#transport-adapters) as an option that also uses `Retry`.

Seems reasonable to use `HTTPAdapter`, since it's already using `requests`. Must also consider streaming interruption, as noted [here](https://github.com/Imageomics/cautious-robot/pull/1#discussion_r1627840598).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.