GCSFileIO: use single-shot upload for small objects below a configurable threshold
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
#### Summary
`GCSFileIO` / `GCSOutputStream` always uploads via GCS `WriteChannel` (`storage.writer(...)`), which uses **resumable uploads**. For small objects this adds unnecessary round-trips and latency compared to a single-shot / simple upload (`storage.create(...)`).
`S3FileIO` already switches between single `PutObject` and multipart based on size . GCS has no equivalent.
#### Motivation / Use case
Iceberg writes many relatively small objects (manifests, metadata JSON, delete files, etc.). For these, resumable-upload setup cost often dominates transfer time.
Google Cloud Storage Java guidance also recommends:
- small content → `storage.create(blobInfo, bytes)` (single request)
- large / streaming content → `storage.writer(...)` (resumable)
Today Iceberg always takes the resumable path, which is suboptimal for small files and creates a feature gap vs S3.
#### Current behavior
In `GCSOutputStream`, uploads always open a write channel:
```java
WriteChannel channel =
storage.writer(
BlobInfo.newBuilder(blobId).build(), writeOptions.toArray(new BlobWriteOption[0]));
gcpProperties.channelWriteChunkSize().ifPresent(channel::setChunkSize);
stream = Channels.newOutputStream(channel);
### Query engine
_No response_
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start with GCSFileIO and GCSOutputStream, then compare the existing storage.writer(...) path with S3FileIO's size-based upload choice. Determine how a configurable threshold should select storage.create(...) for small objects while retaining WriteChannel for large or streaming uploads; done means the threshold is configurable and both upload paths work correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, java
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100