S3 CompleteMultipartUploadRequest serializes pretty XML, causing payload to exceed 1MiB with fewer than 10,000 parts
- Lingua principale
- C++
- Stelle
- 2.2k
- Fork
- 1.2k
- Merge medio
- 4g 11h
- PR unite (30g)
- 13
Descrizione
### Describe the bug
`CompleteMultipartUploadRequest::SerializePayload()` in AWS SDK for C++ serializes the multipart completion body as pretty-printed XML, including indentation and newlines for every `` entry.
For large multipart uploads close to the S3 maximum of 10,000 parts, this extra whitespace can make the `CompleteMultipartUpload` request body exceed 1 MiB. Some S3-compatible storage systems, such as Ceph RGW deployments with a 1 MiB request/XML parsing limit, reject the request even though the upload uses fewer than 10,000 parts.
In our case, an upload with 9,331 parts failed at `CompleteMultipartUpload` with HTTP 416. All `UploadPart` requests succeeded.
## Observed Behavior
The SDK generates XML like this:
```xml
"4f08eef4096726bff1125e1b1b734644"
1
```
For 9,331 parts, the pretty-printed XML is approximately:
```
1,090,743 bytes ~= 1.04 MiB
```
A compact equivalent would be approximately:
```
829,472 bytes ~= 0.79 MiB
```
So the whitespace alone adds about 255 KiB and can determine whether the request succeeds.
### Regression Issue
- [ ] Select this option if this issue appears to be a regression.
### Expected Behavior
The SDK should either:
1. Serialize CompleteMultipartUpload XML in compact form by default, or
2. Provide a configuration option to use compact XML serialization for request payloads.
Compact XML is semantically equivalent and accepted by S3-compatible APIs.
### Current Behavior
`CompleteMultipartUploadRequest::SerializePayload()` calls `XmlDocument::ConvertToString()`, and `XmlDocument::ConvertToString()` uses the default tinyxml2 `XMLPrinter`, which pretty-prints XML. The resulting string is then used directly as the HTTP request body.
Relevant source paths:
- generated/src/aws-cpp-sdk-s3/source/model/CompleteMultipartUploadRequest.cpp
- src/aws-cpp-sdk-core/source/utils/xml/XmlSerializer.cpp
- src/aws-cpp-sdk-core/source/AmazonSerializableWebServiceRequest.cpp
I checked the latest tag 1.11.865, and the behavior appears unchanged.
### Reproduction Steps
1. Create a multipart upload.
2. Upload around 9,000 to 10,000 parts.
3. Build a CompleteMultipartUploadRequest with all returned ETags.
4. Call SerializePayload() and measure the payload size.
5. Compare it with a compact XML representation.
Pseudo-code:
~~~
Aws::S3::Model::CompleteMultipartUploadRequest req;
req.SetBucket(bucket);
req.SetKey(key);
req.SetUploadId(upload_id);
Aws::S3::Model::CompletedMultipartUpload upload;
for (int i = 0; i < etags.size(); ++i) {
Aws::S3::Model::CompletedPart part;
upload.AddParts(part.WithETag(etags[i]).WithPartNumber(i + 1));
}
req.SetMultipartUpload(upload);
auto xml = req.SerializePayload();
std::cout << xml.size() << std::endl;
~~~
## Environment
- AWS SDK for C++ version: observed with 1.11.267; also checked source for 1.11.865
- Service: S3-compatible storage, Ceph RGW
- Operation: CompleteMultipartUpload
- Number of parts: 9,331
- Failure: HTTP 416 from storage service during complete multipart upload
## Why This Matters
S3 allows up to 10,000 parts. With pretty-printed XML, the SDK can exceed a 1 MiB XML/request-body limit before reaching that documented maximum. Compact XML would avoid this for common ETag sizes and improve compatibility with S3-compatible storage systems.
Would the maintainers consider adding compact XML serialization for S3 request payloads, or exposing an option to enable it?
### Possible Solution
_No response_
### Additional Information/Context
_No response_
### AWS CPP SDK version used
1.11.855
### Compiler and Version used
Apple clang version 17.0.0 (clang-1700.0.13.5) Target: arm64-apple-darwin24.5.0
### Operating System and version
MacBook-Pro.local 24.5.0 Darwin Kernel Version 24.5.0
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia leggendo generated/src/aws-cpp-sdk-s3/source/model/CompleteMultipartUploadRequest.cpp, quindi segui XmlDocument::ConvertToString() in src/aws-cpp-sdk-core/source/utils/xml/XmlSerializer.cpp. Riproduci il confronto delle dimensioni del payload con una richiesta contenente circa 9.000–10.000 parts e analizza AmazonSerializableWebServiceRequest.cpp per vedere come viene inviato il body serializzato. Il lavoro è completato quando CompleteMultipartUpload può produrre XML compatto o esporre un'opzione di serializzazione compatta senza modificare la semantica XML.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- cpp
- Ambito
- api
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 57/100