Support configuring custom MetricPublisher for S3FileIO
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 129
Description
### Feature Request / Improvement
We run Spark+Iceberg at various customers. Many of them have on-premise setups with S3 compatible storages. We can bump into a variety of issues that can be complex to debug.
One of the things we have added is capturing metrics emitted by the AWS SDK. We are now working around S3FileIO on our custom fork, but ideally we'd have this in the mainline. All we need is to inject our implementation of the AWS SDK MetricsPublisher interface as part of the configuration
### Query engine
Spark
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
Contributor guide
Research direction
Start at S3FileIO and trace how its configuration creates the AWS SDK client and accepts configurable properties. Check the AWS SDK MetricsPublisher interface and determine how an implementation can be supplied through Iceberg configuration. Done means a configured publisher is used by S3FileIO and the behavior is covered by relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100