S3 upload with default CRC64NVME trailer checksum fails with BadDigest when the request is retried
- Dominant language
- C++
- Stars
- 2.2k
- Forks
- 1.2k
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 12
Description
### Describe the bug
When the SDK sends an S3 `PutObject` with its **default** request checksum (CRC64NVME, `when_supported`), the checksum is transmitted as an HTTP **trailer** over `aws-chunked` encoding. If that request is then **retried**, the retry appears to re-send the body without correctly resetting the `AwsChunkedStream` wrapper state, so the bytes S3 receives no longer correspond to the trailer value. S3 rejects every attempt with:
```
BadDigest: The CRC64NVME you specified did not match the calculated checksum.
```
The failure is **100% deterministic** and **independent of object size** — a ~600 byte text object fails identically to a 2 MB one.
The easiest way to force a guaranteed retry on every request is a signing-region mismatch: configure the client with `region = "aws-global"` (which the S3 endpoint ruleset resolves to the global endpoint with `signingRegion: us-east-1`) while the target bucket lives in any other region. S3 rejects attempt 1 with `AuthorizationHeaderMalformed`, the SDK re-signs and retries against the correct regional endpoint, and that retry then fails with `BadDigest`.
To be clear about scope: `aws-global` against a non-us-east-1 bucket is a misconfiguration on our side, and we have corrected it. We are reporting this because **the retry path itself looks unsafe whenever a trailer checksum is in play.** A wrong-region redirect is merely a reliable trigger; the same reset would presumably be needed after a `503 SlowDown`, a transient `500`, or a dropped connection — all of which happen routinely in production. Before default checksums existed, these retries simply succeeded.
### Expected behavior
A retried `PutObject` should re-send a body whose bytes match the trailer checksum, so the retry succeeds (as it did before default request checksums were introduced).
### Current behavior
Every retried upload fails permanently with `BadDigest`. With a guaranteed-retry trigger, throughput to the bucket is zero — in our case ~1,000 failures/minute with `Uploaded: 0` sustained over 40+ minutes, across object sizes from ~600 bytes to ~2 MB.
### Reproduction Steps
1. Create a bucket in a region other than `us-east-1` (we used `eu-west-1` and `ap-south-1`).
2. Configure an S3 client whose region forces a signing mismatch:
```cpp
Aws::S3::S3ClientConfiguration config;
config.region = "aws-global"; // resolves to global endpoint, signingRegion us-east-1
config.scheme = Aws::Http::Scheme::HTTPS;
Aws::Transfer::TransferManagerConfiguration tmConfig(&executor);
tmConfig.s3Client = Aws::MakeShared("tag", config);
auto transferManager = Aws::Transfer::TransferManager::Create(tmConfig);
// Default checksum behaviour — nothing checksum-related is set anywhere.
auto handle = transferManager->UploadFile(
"/path/to/any/file", "my-bucket-in-eu-west-1", "some/key.ts", "video/mp2t", {});
handle->WaitUntilFinished();
```
3. Every upload fails with `BadDigest: The CRC64NVME you specified did not match the calculated checksum.`
4. Set `config.region = "eu-west-1"` (matching the bucket) and the same code succeeds immediately — no retry, no checksum mismatch.
5. Alternatively keep `region = "aws-global"` and set `AWS_REQUEST_CHECKSUM_CALCULATION=when_required`. Uploads then succeed *despite* the wrong-region retry, because no trailer checksum is sent. This is what points at the trailer/retry interaction rather than the redirect itself.
The wrong-region retry is observable independently (values redacted):
```
attempt 1 → https://.s3.amazonaws.com/...
Authorization: AWS4-HMAC-SHA256 Credential=.../us-east-1/s3/aws4_request
← AuthorizationHeaderMalformed: the region 'us-east-1' is wrong; expecting 'eu-west-1'
x-amz-bucket-region: eu-west-1
attempt 2 → https://.s3.eu-west-1.amazonaws.com/...
Authorization: AWS4-HMAC-SHA256 Credential=.../eu-west-1/s3/aws4_request
```
### Possible Solution
Reset the `AwsChunkedStream` wrapper (in addition to seeking the underlying stream) before a retried request re-sends its body, so the emitted chunk framing and trailer are regenerated for each attempt.
Two previously-reported defects live in the same stream and may be related:
- #3259 — Incorrect chunked encoding with 0-length underlying stream
- #3732 — `AwsChunkedStream`: unbounded `std::stringstream` memory growth causes OOM (confirms this stream buffers into a `std::stringstream` and has had state-management problems)
### Additional Information/Context
What we verified directly:
- The two-pass retry, captured on the wire (shown above).
- 100% failure rate, size-independent, including objects of a few hundred bytes.
- Fixing the region so no retry occurs eliminates the failure entirely — confirmed by an A/B on a single host: two jobs, same process, same SDK, same bucket, same credentials, differing only in the configured region. The `aws-global` job held `Uploaded: 0 / Pending: 1523` and rising; the correctly-configured job ran `Uploaded:` climbing steadily with `Failed: 0 / Pending: 0`.
- An older build of our application bundling a pre-CRC64NVME SDK (no `crc64nvme` symbols present) does **not** exhibit this against the same bucket with the same `aws-global` misconfiguration — the retry there silently succeeds.
What we did **not** verify: we did not capture the raw request bodies of attempt 1 vs attempt 2, so the stream-reset mechanism is our best-supported explanation rather than a directly observed one. Happy to gather more detail if useful.
Our application does not set a checksum algorithm anywhere; the CRC64NVME behaviour is entirely the SDK default.
### AWS CPP SDK version used
1.11.672
### Compiler and Version used
GCC (distribution default for Rocky Linux 9), x86_64
### Operating System and version
Rocky Linux 9.7, x86_64
Contributor guide
Research direction
Start from the AwsChunkedStream implementation and the retry path used by S3 PutObject or TransferManager uploads. Reproduce the two-pass request with a forced signing-region mismatch, then verify that the retried body and CRC64NVME trailer match and the upload succeeds; related issues #3259 and #3732 provide additional stream context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, cpp
- Domain
- api, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100