apache / apache/iceberg

GCP: Support write-time object context annotations in GCS FileIO (equivalent of S3 write.tags)

Open
#16,995 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

## Background

The AWS module supports attaching arbitrary key-value tags to S3 objects at write time via
`s3.write.tags.=` catalog properties (see `S3FileIOProperties` and `S3OutputStream`).
These tags are embedded in the upload request with no extra API call and can drive S3 lifecycle
policies.

GCS has a direct equivalent called **object contexts** — key-value metadata embedded
during the object write. The GCS Java SDK supports this via
`BlobInfo.Builder.setContexts(ObjectContexts)` since
[commit `485aefd`](https://github.com/googleapis/java-storage/commit/485aefd3047c52c98d8bd913033c8aee1473e988)
(google-cloud-storage ≥ 2.57).

## Proposed change

Add `gcs.write.object-context.=` catalog property support, mirroring the S3 design. This would be useful to add additional metadata like environment, team and table info to the catalog files that are created in GCS.

- **`GCPProperties`** — new constant `GCS_WRITE_OBJECT_CONTEXT_PREFIX = "gcs.write.object-context."`;
constructor collects all matching properties into an immutable `Map writeObjectContexts`;
new accessor `writeObjectContexts()`
- **`GCSOutputStream`** — new private `buildBlobInfo(BlobId, GCPProperties)` that calls
`builder.setContexts(ObjectContexts)` when contexts are configured before passing `BlobInfo` to
`storage.writer()`. No changes to the stream mechanics.
- No changes to `GCSFileIO`, `GCSOutputFile`, or any module outside `gcp/`

## Example usage

```properties
spark.sql.catalog.my_catalog.gcs.write.object-context.env=prod
spark.sql.catalog.my_catalog.gcs.write.object-context.team=data-platform
spark.sql.catalog.my_catalog.gcs.write.object-context.iceberg-table=orders
```

## Related

- S3 write-tags: apache/iceberg#4259
- GCS object contexts SDK commit: [[googleapis/java-storage@485aefd](https://github.com/googleapis/java-storage/commit/485aefd3047c52c98d8bd913033c8aee1473e988)](https://github.com/googleapis/java-storage/commit/485aefd3047c52c98d8bd913033c8aee1473e988)
- GCS object context docs: https://cloud.google.com/storage/docs/use-object-contexts

## AI Disclosure
LLM is used to research the details of S3 write tags and draft the issue description.

### Query engine

None

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start by comparing the existing S3 write-tags implementation with GCPProperties and GCSOutputStream in the gcp module. Check that configured gcs.write.object-context. properties are collected and applied to the BlobInfo before storage.writer() is called; done means GCS writes include the configured contexts without changing stream mechanics.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
cloud
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.