allenai / allenai/cached_path

Support for presigned URLs

Ouverte
#252 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Feature request
Langage dominant
Python
Étoiles
47
Forks
22
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

**Is your feature request related to a problem? Please describe.**
I love using this library, but I have a pretty specific problem: Some of the URLs that I'd like to download are presigned R2 Cloudflare urls: https://developers.cloudflare.com/r2/api/s3/presigned-urls/
This routes to the `HTTPClient` which then throws an 403 error when making the head request.

**Describe the solution you'd like**
I'm not sure what is the ideal way to support this, but I currently patched the code to fallback to a get request. It would be nice if something official was supported however.

My current workaround:
```
from cached_path.schemes.http import (
HttpClient,
MaxRetryError,
RecoverableServerError,
session_with_backoff,
)

class HttpClient(HttpClient):
@property
def head_response(self):
"""
Presigned URLs from Cloudflare R2 throw 403 for HEAD requests: https://developers.cloudflare.com/r2/api/s3/presigned-urls/
This is a workaround to be able to avoid a fatal error.
"""
if self._head_response is None:
try:
with session_with_backoff(self.headers) as session:
response = session.head(self.resource, allow_redirects=True)
if response.status_code == 403:
response = session.get(self.resource, allow_redirects=True)
except MaxRetryError as e:
raise RecoverableServerError(e.reason)
self.validate_response(response)
self._head_response = response
return self._head_response
else:
return self._head_response
```

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.