apache / apache/iceberg

Support configuring custom MetricPublisher for S3FileIO

Open
#15,182 2 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 16h
Merged PRs (30d)
129

Description

### Feature Request / Improvement

We run Spark+Iceberg at various customers. Many of them have on-premise setups with S3 compatible storages. We can bump into a variety of issues that can be complex to debug.

One of the things we have added is capturing metrics emitted by the AWS SDK. We are now working around S3FileIO on our custom fork, but ideally we'd have this in the mainline. All we need is to inject our implementation of the AWS SDK MetricsPublisher interface as part of the configuration

### Query engine

Spark

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Contributor guide

Open the contributing guide

Research direction

Start at S3FileIO and trace how its configuration creates the AWS SDK client and accepts configurable properties. Check the AWS SDK MetricsPublisher interface and determine how an implementation can be supplied through Iceberg configuration. Done means a configured publisher is used by S3FileIO and the behavior is covered by relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
cloud
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.