S3 CompleteMultipartUploadRequest serializes pretty XML, causing payload to exceed 1MiB with fewer than 10,000 parts
- Vorherrschende Sprache
- C++
- Sterne
- 2.2k
- Forks
- 1.2k
- Ø Merge
- 4 T. 11 Std.
- Gemergte PRs (30 T.)
- 13
Beschreibung
### Describe the bug
`CompleteMultipartUploadRequest::SerializePayload()` in AWS SDK for C++ serializes the multipart completion body as pretty-printed XML, including indentation and newlines for every `` entry.
For large multipart uploads close to the S3 maximum of 10,000 parts, this extra whitespace can make the `CompleteMultipartUpload` request body exceed 1 MiB. Some S3-compatible storage systems, such as Ceph RGW deployments with a 1 MiB request/XML parsing limit, reject the request even though the upload uses fewer than 10,000 parts.
In our case, an upload with 9,331 parts failed at `CompleteMultipartUpload` with HTTP 416. All `UploadPart` requests succeeded.
## Observed Behavior
The SDK generates XML like this:
```xml
"4f08eef4096726bff1125e1b1b734644"
1
```
For 9,331 parts, the pretty-printed XML is approximately:
```
1,090,743 bytes ~= 1.04 MiB
```
A compact equivalent would be approximately:
```
829,472 bytes ~= 0.79 MiB
```
So the whitespace alone adds about 255 KiB and can determine whether the request succeeds.
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Expected Behavior
The SDK should either:
1. Serialize CompleteMultipartUpload XML in compact form by default, or
2. Provide a configuration option to use compact XML serialization for request payloads.
Compact XML is semantically equivalent and accepted by S3-compatible APIs.
### Current Behavior
`CompleteMultipartUploadRequest::SerializePayload()` calls `XmlDocument::ConvertToString()`, and `XmlDocument::ConvertToString()` uses the default tinyxml2 `XMLPrinter`, which pretty-prints XML. The resulting string is then used directly as the HTTP request body.
Relevant source paths:
- generated/src/aws-cpp-sdk-s3/source/model/CompleteMultipartUploadRequest.cpp
- src/aws-cpp-sdk-core/source/utils/xml/XmlSerializer.cpp
- src/aws-cpp-sdk-core/source/AmazonSerializableWebServiceRequest.cpp
I checked the latest tag 1.11.865, and the behavior appears unchanged.
### Reproduction Steps
1. Create a multipart upload.
2. Upload around 9,000 to 10,000 parts.
3. Build a CompleteMultipartUploadRequest with all returned ETags.
4. Call SerializePayload() and measure the payload size.
5. Compare it with a compact XML representation.
Pseudo-code:
~~~
Aws::S3::Model::CompleteMultipartUploadRequest req;
req.SetBucket(bucket);
req.SetKey(key);
req.SetUploadId(upload_id);
Aws::S3::Model::CompletedMultipartUpload upload;
for (int i = 0; i < etags.size(); ++i) {
Aws::S3::Model::CompletedPart part;
upload.AddParts(part.WithETag(etags[i]).WithPartNumber(i + 1));
}
req.SetMultipartUpload(upload);
auto xml = req.SerializePayload();
std::cout << xml.size() << std::endl;
~~~
## Environment
- AWS SDK for C++ version: observed with 1.11.267; also checked source for 1.11.865
- Service: S3-compatible storage, Ceph RGW
- Operation: CompleteMultipartUpload
- Number of parts: 9,331
- Failure: HTTP 416 from storage service during complete multipart upload
## Why This Matters
S3 allows up to 10,000 parts. With pretty-printed XML, the SDK can exceed a 1 MiB XML/request-body limit before reaching that documented maximum. Compact XML would avoid this for common ETag sizes and improve compatibility with S3-compatible storage systems.
Would the maintainers consider adding compact XML serialization for S3 request payloads, or exposing an option to enable it?
### Possible Solution
_No response_
### Additional Information/Context
_No response_
### AWS CPP SDK version used
1.11.855
### Compiler and Version used
Apple clang version 17.0.0 (clang-1700.0.13.5) Target: arm64-apple-darwin24.5.0
### Operating System and version
MacBook-Pro.local 24.5.0 Darwin Kernel Version 24.5.0
Beitragsleitfaden
Rechercherichtung
Beginne mit dem Lesen von generated/src/aws-cpp-sdk-s3/source/model/CompleteMultipartUploadRequest.cpp und verfolge dann XmlDocument::ConvertToString() in src/aws-cpp-sdk-core/source/utils/xml/XmlSerializer.cpp. Reproduziere den Vergleich der Payload-Größe mit einer Anfrage, die ungefähr 9.000–10.000 Parts enthält, und untersuche AmazonSerializableWebServiceRequest.cpp darauf, wie der serialisierte Body gesendet wird. Als erledigt gilt die Aufgabe, wenn CompleteMultipartUpload kompaktes XML erzeugen kann oder eine Option für die kompakte Serialisierung bereitstellt, ohne die XML-Semantik zu ändern.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- cpp
- Bereich
- api
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 57/100