Skip to content

Support for presigned URLs #252

@vinchg

Description

@vinchg

Is your feature request related to a problem? Please describe.
I love using this library, but I have a pretty specific problem: Some of the URLs that I'd like to download are presigned R2 Cloudflare urls: https://developers.cloudflare.com/r2/api/s3/presigned-urls/
This routes to the HTTPClient which then throws an 403 error when making the head request.

Describe the solution you'd like
I'm not sure what is the ideal way to support this, but I currently patched the code to fallback to a get request. It would be nice if something official was supported however.

My current workaround:

from cached_path.schemes.http import (
    HttpClient,
    MaxRetryError,
    RecoverableServerError,
    session_with_backoff,
)

class HttpClient(HttpClient):
    @property
    def head_response(self):
        """
        Presigned URLs from Cloudflare R2 throw 403 for HEAD requests: https://developers.cloudflare.com/r2/api/s3/presigned-urls/
        This is a workaround to be able to avoid a fatal error.
        """
        if self._head_response is None:
            try:
                with session_with_backoff(self.headers) as session:
                    response = session.head(self.resource, allow_redirects=True)
                    if response.status_code == 403:
                        response = session.get(self.resource, allow_redirects=True)
            except MaxRetryError as e:
                raise RecoverableServerError(e.reason)
            self.validate_response(response)
            self._head_response = response
            return self._head_response
        else:
            return self._head_response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions