blob/s3blob: make it possible to configure the default integrity protection#3634
Conversation
1a17166 to
c69a9c8
Compare
|
Can you do this without updating all of the |
I need |
c69a9c8 to
a266faf
Compare
|
Tests are still failing. It is hard to get them all to pass for contributors when updating dependencies. You can either keep trying, or keep this on hold until I update them. |
a266faf to
1d25529
Compare
|
I just needed to run |
…ection As discussed in aws/aws-sdk-go-v2#3003 and aws/aws-sdk-go-v2#2960, `github.com/aws/aws-sdk-go-v2/service/s3` v1.73.0 changed the AWS SDK default object integrity behavior. Third-party S3 providers, such as Linode, may fail with `XAmzContentSHA256Mismatch` error as a result. A workaround is to set the `AWS_REQUEST_CHECKSUM_CALCULATION` and `AWS_RESPONSE_CHECKSUM_VALIDATION` to `when_required`. However, these environment variables do not affect multipart uploads that used the SDK's `manager.Uploader` implementation (aws/aws-sdk-go-v2#3007). Multipart uploads fail with a 400 Bad Request due to the inclusion of `X-Amz-Sdk-Checksum-Algorithm: CRC32` HTTP headers. With aws/aws-sdk-go-v2#3151, the default integrity protection can be explicitly configured for `manager.Uploader`. To ensure backwards compatibility with third-party S3 providers, this commit adds support for two query parameters: * `request_checksum_calculation` - `when_supported`, `when_required` * `response_checksum_calculation` - `when_supported`, `when_required` For example, on Linode, the defaults don't work: ``` % cat main.go | ./gocdk-blob upload "s3://smybucket?endpoint=https://us-sea-1.linodeobjects.com®ion=us-sea-1" main.go gocdk-blob: closing the writer: blob (key "main.go") (code=Unknown): operation error S3: PutObject, https response error StatusCode: 400, RequestID: <redacted>, HostID: <redacted>, api error XAmzContentSHA256Mismatch: UnknownError ``` Using `request_checksum_calculation=when_required` works: ``` % cat main.go | ./gocdk-blob upload "s3://smybucket?endpoint=https://us-sea-1.linodeobjects.com®ion=us-sea-1&request_checksum_calculation=when_required" main.go % ``` This test was repeated with a larger to validate multipart uploads work.
1d25529 to
1beea01
Compare
As discussed in aws/aws-sdk-go-v2#3003 and aws/aws-sdk-go-v2#2960,
github.com/aws/aws-sdk-go-v2/service/s3v1.73.0 changed the AWS SDK default object integrity behavior. Third-party S3 providers, such as Linode, may fail withXAmzContentSHA256Mismatcherror as a result.A workaround is to set the
AWS_REQUEST_CHECKSUM_CALCULATIONandAWS_RESPONSE_CHECKSUM_VALIDATIONtowhen_required.However, these environment variables do not affect multipart uploads that used the SDK's
manager.Uploaderimplementation. Multipart uploads fail with a 400 Bad Request due to the inclusion ofX-Amz-Sdk-Checksum-Algorithm: CRC32HTTP headers. With aws/aws-sdk-go-v2#3151, the default integrity protection can be configured explicitly formanager.Uploader.To ensure backwards compatibility with third-party S3 providers, this commit adds support for two query parameters:
request_checksum_calculation-when_supported,when_requiredresponse_checksum_calculation-when_supported,when_requiredFor example, on Linode, the defaults don't work:
Using
request_checksum_calculation=when_requiredworks:This test was repeated with a larger to validate multipart uploads work.