apache / apache/iceberg

GCSFileIO: use single-shot upload for small objects below a configurable threshold

Open
#17,524 1 comment 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

#### Summary
`GCSFileIO` / `GCSOutputStream` always uploads via GCS `WriteChannel` (`storage.writer(...)`), which uses **resumable uploads**. For small objects this adds unnecessary round-trips and latency compared to a single-shot / simple upload (`storage.create(...)`).

`S3FileIO` already switches between single `PutObject` and multipart based on size . GCS has no equivalent.

#### Motivation / Use case
Iceberg writes many relatively small objects (manifests, metadata JSON, delete files, etc.). For these, resumable-upload setup cost often dominates transfer time.

Google Cloud Storage Java guidance also recommends:
- small content → `storage.create(blobInfo, bytes)` (single request)
- large / streaming content → `storage.writer(...)` (resumable)

Today Iceberg always takes the resumable path, which is suboptimal for small files and creates a feature gap vs S3.

#### Current behavior
In `GCSOutputStream`, uploads always open a write channel:

```java
WriteChannel channel =
storage.writer(
BlobInfo.newBuilder(blobId).build(), writeOptions.toArray(new BlobWriteOption[0]));
gcpProperties.channelWriteChunkSize().ifPresent(channel::setChunkSize);
stream = Channels.newOutputStream(channel);

### Query engine

_No response_

### Willingness to contribute

- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start with GCSFileIO and GCSOutputStream, then compare the existing storage.writer(...) path with S3FileIO's size-based upload choice. Determine how a configurable threshold should select storage.create(...) for small objects while retaining WriteChannel for large or streaming uploads; done means the threshold is configurable and both upload paths work correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.