Is your feature request related to a problem? Please describe.
I love using this library, but I have a pretty specific problem: Some of the URLs that I'd like to download are presigned R2 Cloudflare urls: https://developers.cloudflare.com/r2/api/s3/presigned-urls/
This routes to the HTTPClient which then throws an 403 error when making the head request.
Describe the solution you'd like
I'm not sure what is the ideal way to support this, but I currently patched the code to fallback to a get request. It would be nice if something official was supported however.
My current workaround:
from cached_path.schemes.http import (
HttpClient,
MaxRetryError,
RecoverableServerError,
session_with_backoff,
)
class HttpClient(HttpClient):
@property
def head_response(self):
"""
Presigned URLs from Cloudflare R2 throw 403 for HEAD requests: https://developers.cloudflare.com/r2/api/s3/presigned-urls/
This is a workaround to be able to avoid a fatal error.
"""
if self._head_response is None:
try:
with session_with_backoff(self.headers) as session:
response = session.head(self.resource, allow_redirects=True)
if response.status_code == 403:
response = session.get(self.resource, allow_redirects=True)
except MaxRetryError as e:
raise RecoverableServerError(e.reason)
self.validate_response(response)
self._head_response = response
return self._head_response
else:
return self._head_response
Is your feature request related to a problem? Please describe.
I love using this library, but I have a pretty specific problem: Some of the URLs that I'd like to download are presigned R2 Cloudflare urls: https://developers.cloudflare.com/r2/api/s3/presigned-urls/
This routes to the
HTTPClientwhich then throws an 403 error when making the head request.Describe the solution you'd like
I'm not sure what is the ideal way to support this, but I currently patched the code to fallback to a get request. It would be nice if something official was supported however.
My current workaround: