Skip to content

blob/s3blob: make it possible to configure the default integrity protection#3634

Merged
vangent merged 2 commits intogoogle:masterfrom
stanhu:sh-s3-configure-checksum-validation
Nov 5, 2025
Merged

blob/s3blob: make it possible to configure the default integrity protection#3634
vangent merged 2 commits intogoogle:masterfrom
stanhu:sh-s3-configure-checksum-validation

Conversation

@stanhu
Copy link
Contributor

@stanhu stanhu commented Nov 5, 2025

As discussed in aws/aws-sdk-go-v2#3003 and aws/aws-sdk-go-v2#2960, github.com/aws/aws-sdk-go-v2/service/s3 v1.73.0 changed the AWS SDK default object integrity behavior. Third-party S3 providers, such as Linode, may fail with XAmzContentSHA256Mismatch error as a result.

A workaround is to set the AWS_REQUEST_CHECKSUM_CALCULATION and AWS_RESPONSE_CHECKSUM_VALIDATION to when_required.

However, these environment variables do not affect multipart uploads that used the SDK's manager.Uploader implementation. Multipart uploads fail with a 400 Bad Request due to the inclusion of X-Amz-Sdk-Checksum-Algorithm: CRC32 HTTP headers. With aws/aws-sdk-go-v2#3151, the default integrity protection can be configured explicitly for manager.Uploader.

To ensure backwards compatibility with third-party S3 providers, this commit adds support for two query parameters:

  • request_checksum_calculation - when_supported, when_required
  • response_checksum_calculation - when_supported, when_required

For example, on Linode, the defaults don't work:

% cat main.go | ./gocdk-blob upload "s3://smybucket?endpoint=https://us-sea-1.linodeobjects.com&region=us-sea-1" main.go
gocdk-blob: closing the writer: blob (key "main.go") (code=Unknown): operation error S3: PutObject, https response error StatusCode: 400, RequestID: <redacted>, HostID: <redacted>, api error XAmzContentSHA256Mismatch: UnknownError

Using request_checksum_calculation=when_required works:

% cat main.go | ./gocdk-blob upload "s3://smybucket?endpoint=https://us-sea-1.linodeobjects.com&region=us-sea-1&request_checksum_calculation=when_required" main.go
%

This test was repeated with a larger to validate multipart uploads work.

@stanhu stanhu force-pushed the sh-s3-configure-checksum-validation branch 3 times, most recently from 1a17166 to c69a9c8 Compare November 5, 2025 00:25
@vangent
Copy link
Contributor

vangent commented Nov 5, 2025

Can you do this without updating all of the go.mod / go.sum files? Updating to newer dependencies seems like it should be separable (and it's hard for a contributor to get changes like that so that they pass tests).

@stanhu
Copy link
Contributor Author

stanhu commented Nov 5, 2025

Can you do this without updating all of the go.mod / go.sum files? Updating to newer dependencies seems like it should be separable (and it's hard for a contributor to get changes like that so that they pass tests).

I need github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.19.0 at least for aws/aws-sdk-go-v2#3151. Currently this project is using v1.17.84.

@stanhu stanhu force-pushed the sh-s3-configure-checksum-validation branch from c69a9c8 to a266faf Compare November 5, 2025 04:54
@vangent
Copy link
Contributor

vangent commented Nov 5, 2025

Tests are still failing. It is hard to get them all to pass for contributors when updating dependencies. You can either keep trying, or keep this on hold until I update them.

@stanhu stanhu force-pushed the sh-s3-configure-checksum-validation branch from a266faf to 1d25529 Compare November 5, 2025 18:20
@stanhu
Copy link
Contributor Author

stanhu commented Nov 5, 2025

I just needed to run ./internal/testing/gomodcleanup.sh. I hope that should do it.

…ection

As discussed in aws/aws-sdk-go-v2#3003 and
aws/aws-sdk-go-v2#2960,
`github.com/aws/aws-sdk-go-v2/service/s3` v1.73.0 changed the AWS SDK
default object integrity behavior. Third-party S3 providers, such as
Linode, may fail with `XAmzContentSHA256Mismatch` error as a result.

A workaround is to set the `AWS_REQUEST_CHECKSUM_CALCULATION`
and `AWS_RESPONSE_CHECKSUM_VALIDATION` to `when_required`.

However, these environment variables do not affect multipart uploads
that used the SDK's `manager.Uploader` implementation
(aws/aws-sdk-go-v2#3007). Multipart uploads
fail with a 400 Bad Request due to the inclusion of
`X-Amz-Sdk-Checksum-Algorithm: CRC32` HTTP headers. With
aws/aws-sdk-go-v2#3151, the default integrity
protection can be explicitly configured for `manager.Uploader`.

To ensure backwards compatibility with third-party S3 providers, this
commit adds support for two query parameters:

* `request_checksum_calculation` - `when_supported`, `when_required`
* `response_checksum_calculation` - `when_supported`, `when_required`

For example, on Linode, the defaults don't work:

```
% cat main.go | ./gocdk-blob upload "s3://smybucket?endpoint=https://us-sea-1.linodeobjects.com&region=us-sea-1" main.go
gocdk-blob: closing the writer: blob (key "main.go") (code=Unknown): operation error S3: PutObject, https response error StatusCode: 400, RequestID: <redacted>, HostID: <redacted>, api error XAmzContentSHA256Mismatch: UnknownError
```

Using `request_checksum_calculation=when_required` works:

```
% cat main.go | ./gocdk-blob upload "s3://smybucket?endpoint=https://us-sea-1.linodeobjects.com&region=us-sea-1&request_checksum_calculation=when_required" main.go
%
```

This test was repeated with a larger to validate multipart uploads work.
@stanhu stanhu force-pushed the sh-s3-configure-checksum-validation branch from 1d25529 to 1beea01 Compare November 5, 2025 18:24
@vangent vangent merged commit f1d2dbc into google:master Nov 5, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants