googleapis / googleapis/google-cloud-swift
[storage]: support randomized and configurable chunk sizes in StorageW1R3
- Dominant language
- Swift
- Stars
- 26
- Forks
- 10
- Avg merge
- 12h 56m
- Merged PRs (30d)
- 211
Description
# test(storage): support randomized and configurable chunk sizes in `StorageW1R3`
## Problem Description
Currently in `Tests/StorageW1R3/StorageOperations.swift`, resumable uploads hardcode the chunk size to 32 MiB:
```swift
if isResumable {
$0.chunkSize = 32 * 1024 * 1024
$0.resumableUploadThreshold = buffer.readableBytes
}
```
There are no command-line flags in `StorageW1R3` to configure or vary this parameter. As a result:
1. The benchmark cannot evaluate performance under the default SDK chunk size (8 MiB) vs larger chunks (16 MiB, 32 MiB, 64 MiB).
2. The benchmark cannot test single-request streaming (unbuffered resumable uploads, where `chunkSize` is unset).
3. The benchmark cannot evaluate realistic production workloads where chunk sizes vary across operations or explore sensitivity curves between chunk size and network throughput.
4. Hardcoding a single chunk size introduces systematic bias into long-running benchmark runs.
---
## Proposed Solution
Enhance `StorageW1R3` to support flexible and randomized chunk size selection across iterations:
### 1. Command-Line Configuration Options
Add options to `StorageW1R3.swift`:
- `--min-chunk-size` (default: e.g. `8MiB`): Minimum chunk size when randomizing in a range.
- `--max-chunk-size` (default: e.g. `32MiB`): Maximum chunk size when randomizing in a range.
- `--chunk-size-quantum` (default: `256KiB`): Alignment quantum for generated chunk sizes (Google Cloud Storage requires resumable chunks to be multiples of 256 KiB).
- `--chunk-sizes`: A comma-separated list of candidate chunk sizes (e.g. `--chunk-sizes 8MiB,16MiB,32MiB,64MiB`), where each iteration picks one uniformly at random.
- `--chunk-size`: A fixed chunk size override for targeted benchmarks (e.g. `--chunk-size 32MiB`).
### 2. Randomization Logic in `BenchmarkRunner.swift`
For each iteration, pick a chunk size according to the configured strategy:
- **List mode**: Pick uniformly from `--chunk-sizes`.
- **Range mode with quantum**:
```swift
let quantum = chunkSizeQuantum // e.g. 256 * 1024
let minSteps = minChunkSize / quantum
let maxSteps = maxChunkSize / quantum
let selectedSteps = Int.random(in: minSteps...maxSteps)
let iterationChunkSize = selectedSteps * quantum
```
- **Fixed mode**: Use `--chunk-size`.
### 3. Pass Configured Chunk Size to `StorageOperations.upload`
Pass the selected chunk size to `StorageOperations.upload`:
```swift
if isResumable {
$0.chunkSize = iterationChunkSize
$0.resumableUploadThreshold = buffer.readableBytes
}
```
### 4. Record Chunk Size in Benchmark Output
Record the chunk size in `Sample.swift`:
- Either add a new column `ChunkSize` to `Sample.header` and `Sample.toRow()`, or include `chunkSize=` in the `Details` column so existing BigQuery ingest pipelines remain backward-compatible while capturing chunk size dimensions for analysis.
---
## Acceptance Criteria
- Running `StorageW1R3 --min-chunk-size 8MiB --max-chunk-size 64MiB --chunk-size-quantum 8MiB` selects a random multiple of 8 MiB in `[8MiB, 64MiB]` for each resumable upload iteration.
- Running `StorageW1R3 --chunk-sizes 8MiB,32MiB` randomly alternates between 8 MiB and 32 MiB chunk sizes.
- Running `StorageW1R3 --chunk-size 32MiB` preserves existing fixed 32 MiB behavior.
- Chunk sizes are verified to adhere to GCS 256 KiB alignment rules.
- Results emitted to stdout allow filtering and grouping throughput metrics by chunk size.
Contributor guide
Research direction
Start with Tests/StorageW1R3/StorageOperations.swift and the StorageW1R3 entry point in StorageW1R3.swift, then trace how BenchmarkRunner.swift selects upload parameters and how Sample.swift formats output. Run the listed StorageW1R3 command examples; done means fixed, list, and range modes select aligned chunk sizes per iteration and stdout identifies the chunk size for analysis.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- swift
- Domain
- performance, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100