aws / aws/aws-sdk-cpp

S3 CompleteMultipartUploadRequest serializes pretty XML, causing payload to exceed 1MiB with fewer than 10,000 parts

Open
#3,891 1 comment 0 reactions 0 assignees View on GitHub
feature-request
Dominant language
C++
Stars
2.2k
Forks
1.2k
Avg merge
3d 14h
Merged PRs (30d)
12

Description

### Describe the bug

`CompleteMultipartUploadRequest::SerializePayload()` in AWS SDK for C++ serializes the multipart completion body as pretty-printed XML, including indentation and newlines for every `` entry.

For large multipart uploads close to the S3 maximum of 10,000 parts, this extra whitespace can make the `CompleteMultipartUpload` request body exceed 1 MiB. Some S3-compatible storage systems, such as Ceph RGW deployments with a 1 MiB request/XML parsing limit, reject the request even though the upload uses fewer than 10,000 parts.

In our case, an upload with 9,331 parts failed at `CompleteMultipartUpload` with HTTP 416. All `UploadPart` requests succeeded.

## Observed Behavior

The SDK generates XML like this:

```xml


"4f08eef4096726bff1125e1b1b734644"
1

```
For 9,331 parts, the pretty-printed XML is approximately:
```
1,090,743 bytes ~= 1.04 MiB
```
A compact equivalent would be approximately:
```
829,472 bytes ~= 0.79 MiB
```
So the whitespace alone adds about 255 KiB and can determine whether the request succeeds.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

The SDK should either:
1. Serialize CompleteMultipartUpload XML in compact form by default, or
2. Provide a configuration option to use compact XML serialization for request payloads.
Compact XML is semantically equivalent and accepted by S3-compatible APIs.

### Current Behavior

`CompleteMultipartUploadRequest::SerializePayload()` calls `XmlDocument::ConvertToString()`, and `XmlDocument::ConvertToString()` uses the default tinyxml2 `XMLPrinter`, which pretty-prints XML. The resulting string is then used directly as the HTTP request body.
Relevant source paths:
- generated/src/aws-cpp-sdk-s3/source/model/CompleteMultipartUploadRequest.cpp
- src/aws-cpp-sdk-core/source/utils/xml/XmlSerializer.cpp
- src/aws-cpp-sdk-core/source/AmazonSerializableWebServiceRequest.cpp
I checked the latest tag 1.11.865, and the behavior appears unchanged.

### Reproduction Steps

1. Create a multipart upload.
2. Upload around 9,000 to 10,000 parts.
3. Build a CompleteMultipartUploadRequest with all returned ETags.
4. Call SerializePayload() and measure the payload size.
5. Compare it with a compact XML representation.

Pseudo-code:
~~~
Aws::S3::Model::CompleteMultipartUploadRequest req;
req.SetBucket(bucket);
req.SetKey(key);
req.SetUploadId(upload_id);

Aws::S3::Model::CompletedMultipartUpload upload;
for (int i = 0; i < etags.size(); ++i) {
Aws::S3::Model::CompletedPart part;
upload.AddParts(part.WithETag(etags[i]).WithPartNumber(i + 1));
}
req.SetMultipartUpload(upload);

auto xml = req.SerializePayload();
std::cout << xml.size() << std::endl;
~~~

## Environment
- AWS SDK for C++ version: observed with 1.11.267; also checked source for 1.11.865
- Service: S3-compatible storage, Ceph RGW
- Operation: CompleteMultipartUpload
- Number of parts: 9,331
- Failure: HTTP 416 from storage service during complete multipart upload

## Why This Matters
S3 allows up to 10,000 parts. With pretty-printed XML, the SDK can exceed a 1 MiB XML/request-body limit before reaching that documented maximum. Compact XML would avoid this for common ETag sizes and improve compatibility with S3-compatible storage systems.
Would the maintainers consider adding compact XML serialization for S3 request payloads, or exposing an option to enable it?

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### AWS CPP SDK version used

1.11.855

### Compiler and Version used

Apple clang version 17.0.0 (clang-1700.0.13.5) Target: arm64-apple-darwin24.5.0

### Operating System and version

MacBook-Pro.local 24.5.0 Darwin Kernel Version 24.5.0

Contributor guide

Open the contributing guide

Research direction

Start by reading generated/src/aws-cpp-sdk-s3/source/model/CompleteMultipartUploadRequest.cpp and then trace XmlDocument::ConvertToString() in src/aws-cpp-sdk-core/source/utils/xml/XmlSerializer.cpp. Reproduce the payload-size comparison with a request containing roughly 9,000–10,000 parts, and inspect AmazonSerializableWebServiceRequest.cpp for how the serialized body is sent. Done means CompleteMultipartUpload can produce compact XML or expose a compact-serialization option without changing XML semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
57/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.