aws / aws/aws-sdk-cpp

S3 upload with default CRC64NVME trailer checksum fails with BadDigest when the request is retried

Closed
#3,903 4 comments 0 reactions 0 assignees View on GitHub
bug closing-soon response-requested
Dominant language
C++
Stars
2.2k
Forks
1.2k
Avg merge
3d 14h
Merged PRs (30d)
12

Description

### Describe the bug

When the SDK sends an S3 `PutObject` with its **default** request checksum (CRC64NVME, `when_supported`), the checksum is transmitted as an HTTP **trailer** over `aws-chunked` encoding. If that request is then **retried**, the retry appears to re-send the body without correctly resetting the `AwsChunkedStream` wrapper state, so the bytes S3 receives no longer correspond to the trailer value. S3 rejects every attempt with:

```
BadDigest: The CRC64NVME you specified did not match the calculated checksum.
```

The failure is **100% deterministic** and **independent of object size** — a ~600 byte text object fails identically to a 2 MB one.

The easiest way to force a guaranteed retry on every request is a signing-region mismatch: configure the client with `region = "aws-global"` (which the S3 endpoint ruleset resolves to the global endpoint with `signingRegion: us-east-1`) while the target bucket lives in any other region. S3 rejects attempt 1 with `AuthorizationHeaderMalformed`, the SDK re-signs and retries against the correct regional endpoint, and that retry then fails with `BadDigest`.

To be clear about scope: `aws-global` against a non-us-east-1 bucket is a misconfiguration on our side, and we have corrected it. We are reporting this because **the retry path itself looks unsafe whenever a trailer checksum is in play.** A wrong-region redirect is merely a reliable trigger; the same reset would presumably be needed after a `503 SlowDown`, a transient `500`, or a dropped connection — all of which happen routinely in production. Before default checksums existed, these retries simply succeeded.

### Expected behavior

A retried `PutObject` should re-send a body whose bytes match the trailer checksum, so the retry succeeds (as it did before default request checksums were introduced).

### Current behavior

Every retried upload fails permanently with `BadDigest`. With a guaranteed-retry trigger, throughput to the bucket is zero — in our case ~1,000 failures/minute with `Uploaded: 0` sustained over 40+ minutes, across object sizes from ~600 bytes to ~2 MB.

### Reproduction Steps

1. Create a bucket in a region other than `us-east-1` (we used `eu-west-1` and `ap-south-1`).
2. Configure an S3 client whose region forces a signing mismatch:

```cpp
Aws::S3::S3ClientConfiguration config;
config.region = "aws-global"; // resolves to global endpoint, signingRegion us-east-1
config.scheme = Aws::Http::Scheme::HTTPS;

Aws::Transfer::TransferManagerConfiguration tmConfig(&executor);
tmConfig.s3Client = Aws::MakeShared("tag", config);
auto transferManager = Aws::Transfer::TransferManager::Create(tmConfig);

// Default checksum behaviour — nothing checksum-related is set anywhere.
auto handle = transferManager->UploadFile(
"/path/to/any/file", "my-bucket-in-eu-west-1", "some/key.ts", "video/mp2t", {});
handle->WaitUntilFinished();
```

3. Every upload fails with `BadDigest: The CRC64NVME you specified did not match the calculated checksum.`
4. Set `config.region = "eu-west-1"` (matching the bucket) and the same code succeeds immediately — no retry, no checksum mismatch.
5. Alternatively keep `region = "aws-global"` and set `AWS_REQUEST_CHECKSUM_CALCULATION=when_required`. Uploads then succeed *despite* the wrong-region retry, because no trailer checksum is sent. This is what points at the trailer/retry interaction rather than the redirect itself.

The wrong-region retry is observable independently (values redacted):

```
attempt 1 → https://.s3.amazonaws.com/...
Authorization: AWS4-HMAC-SHA256 Credential=.../us-east-1/s3/aws4_request
← AuthorizationHeaderMalformed: the region 'us-east-1' is wrong; expecting 'eu-west-1'
x-amz-bucket-region: eu-west-1

attempt 2 → https://.s3.eu-west-1.amazonaws.com/...
Authorization: AWS4-HMAC-SHA256 Credential=.../eu-west-1/s3/aws4_request
```

### Possible Solution

Reset the `AwsChunkedStream` wrapper (in addition to seeking the underlying stream) before a retried request re-sends its body, so the emitted chunk framing and trailer are regenerated for each attempt.

Two previously-reported defects live in the same stream and may be related:

- #3259 — Incorrect chunked encoding with 0-length underlying stream
- #3732 — `AwsChunkedStream`: unbounded `std::stringstream` memory growth causes OOM (confirms this stream buffers into a `std::stringstream` and has had state-management problems)

### Additional Information/Context

What we verified directly:

- The two-pass retry, captured on the wire (shown above).
- 100% failure rate, size-independent, including objects of a few hundred bytes.
- Fixing the region so no retry occurs eliminates the failure entirely — confirmed by an A/B on a single host: two jobs, same process, same SDK, same bucket, same credentials, differing only in the configured region. The `aws-global` job held `Uploaded: 0 / Pending: 1523` and rising; the correctly-configured job ran `Uploaded:` climbing steadily with `Failed: 0 / Pending: 0`.
- An older build of our application bundling a pre-CRC64NVME SDK (no `crc64nvme` symbols present) does **not** exhibit this against the same bucket with the same `aws-global` misconfiguration — the retry there silently succeeds.

What we did **not** verify: we did not capture the raw request bodies of attempt 1 vs attempt 2, so the stream-reset mechanism is our best-supported explanation rather than a directly observed one. Happy to gather more detail if useful.

Our application does not set a checksum algorithm anywhere; the CRC64NVME behaviour is entirely the SDK default.

### AWS CPP SDK version used

1.11.672

### Compiler and Version used

GCC (distribution default for Rocky Linux 9), x86_64

### Operating System and version

Rocky Linux 9.7, x86_64

Contributor guide

Open the contributing guide

Research direction

Start from the AwsChunkedStream implementation and the retry path used by S3 PutObject or TransferManager uploads. Reproduce the two-pass request with a forced signing-region mismatch, then verify that the retried body and CRC64NVME trailer match and the upload succeeds; related issues #3259 and #3732 provide additional stream context.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp
Domain
api, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.