googleapis / googleapis/google-cloud-swift
storage: optimize `MultipartUploadStream` for high-throughput uploads
- Dominant language
- Swift
- Stars
- 26
- Forks
- 10
- Avg merge
- 12h 56m
- Merged PRs (30d)
- 211
Description
Gemini is too verbose, but maybe this can be used as context for the fix too
----
# storage: optimize `MultipartUploadStream` for high-throughput uploads
## Problem Description
In `pkgs/swift-google-cloud-storage/Sources/GoogleCloudStorage/MultipartUploadStream.swift`, simple multipart uploads (`performSimpleUpload`) experience severe throughput degradation on large objects, measuring under 55 MiB/s even on high-bandwidth networks.
There are two primary architectural bottlenecks in the current implementation:
### 1. 64 KiB Micro-Chunking During Checksum Calculation
The GCS simple upload endpoint requires the `x-goog-hash` header to be sent in the initial HTTP request headers before the body is received. To compute this header, `MultipartUploadStream.prepare` inspects seekable sources before streaming:
```swift
if var seekable = source as? (any SeekableUploadSource) {
if !autoCalculators.isEmpty {
while let chunk = try await seekable.read(maxBytes: chunkSize) { // chunkSize defaults to 64 KiB
for i in calculators.indices {
calculators[i].update(chunk)
}
}
}
try await seekable.seek(to: 0)
preparedSource = seekable
}
```
Because `chunkSize` defaults to `64 * 1024` (64 KiB):
- A **32 MiB** upload executes **512** sequential `try await seekable.read(maxBytes: 64KB)` calls and 512 `update()` loops before seeking back to 0.
- A **128 MiB** upload executes **2,048** sequential async reads and loops.
- A **256 MiB** upload executes **4,096** sequential async reads and loops.
For in-memory sources (`BytesSource`), the buffer is already contiguously in memory. Slicing it into thousands of 64 KiB buffers and awaiting across async boundaries adds significant CPU and ARC overhead.
### 2. 64 KiB Micro-Chunking During Body Streaming
During transmission, `MultipartUploadStream.AsyncIterator.next()` yields the request body in the same 64 KiB chunks:
```swift
case .body:
let chunk = try await source.read(maxBytes: chunkSize) // 64 KiB
if let chunk = chunk, !chunk.isEmpty {
bytesYielded += UInt64(chunk.count)
return chunk.byteBuffer
}
```
Streaming a 128 MiB body in 64 KiB chunks incurs **2,048** async iterator suspensions and resumptions. Each tiny buffer travels through Swift async task scheduling and the SwiftNIO channel pipeline, starving the socket and preventing the client from reaching line rate (> 90 MiB/s).
In benchmarks like `StorageW1R3` (which randomly selects between resumable and simple uploads via `Bool.random()`), this ~40–55 MiB/s ceiling on simple uploads drags down the overall benchmark score.
---
## Proposed Solution
1. **Fast Contiguous Checksum Calculation for In-Memory Sources**:
When the source is a `BytesSource` (or exposes contiguous memory via `withUnsafeBytes`), compute CRC32C in a single pass over the entire buffer via `_CRC32C.compute(...)`:
```swift
if let bytesSource = source as? BytesSource {
let crc = bytesSource.buffer.withUnsafeBytes { _CRC32C.compute($0) }
// populate CRC32C header without sequential read loop
}
```
This takes ~4 ms for 32 MiB using hardware SSE4.2 instructions, rather than hundreds of async read calls.
2. **Increase Streaming Chunk Size**:
Increase the default streaming `chunkSize` in `MultipartUploadStream` from 64 KiB to **1 MiB or 2 MiB** (`2 * 1024 * 1024`).
- For a 128 MiB upload, this reduces async iterator context switches from 2,048 down to 64 (a 32x reduction).
- This allows SwiftNIO and the TCP stack to keep socket buffers saturated without task starvation.
3. **Configurable Streaming Chunk Size**:
Allow `chunkSize` to be configured via `UploadOptions` or use an internal constant optimized for high-throughput streaming.
---
## Acceptance Criteria
- Simple uploads of 32 MiB, 64 MiB, and 128 MiB objects achieve > 90 MiB/s throughput on gigabit+ networks.
- In-memory `BytesSource` uploads compute CRC32C in a single call without multi-pass slicing.
- All existing multipart upload tests (`MultipartUploadStreamTests`, `SimpleUploadTests`) pass with `-warnings-as-errors`.
Contributor guide
Research direction
Start in pkgs/swift-google-cloud-storage/Sources/GoogleCloudStorage/MultipartUploadStream.swift, reading prepare and AsyncIterator.next() to understand checksum calculation and body chunking. Then inspect MultipartUploadStreamTests and SimpleUploadTests, and verify the stated throughput and CRC32C acceptance criteria without breaking existing uploads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- cloud, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100