S3 CompleteMultipartUploadRequest serializes pretty XML, causing payload to exceed 1MiB with fewer than 10,000 parts
- Dominant language
- C++
- Stars
- 2.2k
- Forks
- 1.2k
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 12
Description
### Describe the bug
`CompleteMultipartUploadRequest::SerializePayload()` in AWS SDK for C++ serializes the multipart completion body as pretty-printed XML, including indentation and newlines for every `` entry.
For large multipart uploads close to the S3 maximum of 10,000 parts, this extra whitespace can make the `CompleteMultipartUpload` request body exceed 1 MiB. Some S3-compatible storage systems, such as Ceph RGW deployments with a 1 MiB request/XML parsing limit, reject the request even though the upload uses fewer than 10,000 parts.
In our case, an upload with 9,331 parts failed at `CompleteMultipartUpload` with HTTP 416. All `UploadPart` requests succeeded.
## Observed Behavior
The SDK generates XML like this:
```xml
"4f08eef4096726bff1125e1b1b734644"
1
```
For 9,331 parts, the pretty-printed XML is approximately:
```
1,090,743 bytes ~= 1.04 MiB
```
A compact equivalent would be approximately:
```
829,472 bytes ~= 0.79 MiB
```
So the whitespace alone adds about 255 KiB and can determine whether the request succeeds.
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Expected Behavior
The SDK should either:
1. Serialize CompleteMultipartUpload XML in compact form by default, or
2. Provide a configuration option to use compact XML serialization for request payloads.
Compact XML is semantically equivalent and accepted by S3-compatible APIs.
### Current Behavior
`CompleteMultipartUploadRequest::SerializePayload()` calls `XmlDocument::ConvertToString()`, and `XmlDocument::ConvertToString()` uses the default tinyxml2 `XMLPrinter`, which pretty-prints XML. The resulting string is then used directly as the HTTP request body.
Relevant source paths:
- generated/src/aws-cpp-sdk-s3/source/model/CompleteMultipartUploadRequest.cpp
- src/aws-cpp-sdk-core/source/utils/xml/XmlSerializer.cpp
- src/aws-cpp-sdk-core/source/AmazonSerializableWebServiceRequest.cpp
I checked the latest tag 1.11.865, and the behavior appears unchanged.
### Reproduction Steps
1. Create a multipart upload.
2. Upload around 9,000 to 10,000 parts.
3. Build a CompleteMultipartUploadRequest with all returned ETags.
4. Call SerializePayload() and measure the payload size.
5. Compare it with a compact XML representation.
Pseudo-code:
~~~
Aws::S3::Model::CompleteMultipartUploadRequest req;
req.SetBucket(bucket);
req.SetKey(key);
req.SetUploadId(upload_id);
Aws::S3::Model::CompletedMultipartUpload upload;
for (int i = 0; i < etags.size(); ++i) {
Aws::S3::Model::CompletedPart part;
upload.AddParts(part.WithETag(etags[i]).WithPartNumber(i + 1));
}
req.SetMultipartUpload(upload);
auto xml = req.SerializePayload();
std::cout << xml.size() << std::endl;
~~~
## Environment
- AWS SDK for C++ version: observed with 1.11.267; also checked source for 1.11.865
- Service: S3-compatible storage, Ceph RGW
- Operation: CompleteMultipartUpload
- Number of parts: 9,331
- Failure: HTTP 416 from storage service during complete multipart upload
## Why This Matters
S3 allows up to 10,000 parts. With pretty-printed XML, the SDK can exceed a 1 MiB XML/request-body limit before reaching that documented maximum. Compact XML would avoid this for common ETag sizes and improve compatibility with S3-compatible storage systems.
Would the maintainers consider adding compact XML serialization for S3 request payloads, or exposing an option to enable it?
### Possible Solution
_No response_
### Additional Information/Context
_No response_
### AWS CPP SDK version used
1.11.855
### Compiler and Version used
Apple clang version 17.0.0 (clang-1700.0.13.5) Target: arm64-apple-darwin24.5.0
### Operating System and version
MacBook-Pro.local 24.5.0 Darwin Kernel Version 24.5.0
Contributor guide
Research direction
Start by reading generated/src/aws-cpp-sdk-s3/source/model/CompleteMultipartUploadRequest.cpp and then trace XmlDocument::ConvertToString() in src/aws-cpp-sdk-core/source/utils/xml/XmlSerializer.cpp. Reproduce the payload-size comparison with a request containing roughly 9,000–10,000 parts, and inspect AmazonSerializableWebServiceRequest.cpp for how the serialized body is sent. Done means CompleteMultipartUpload can produce compact XML or expose a compact-serialization option without changing XML semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- api
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 57/100