apache / apache/iceberg

Avoid loading all AWS client dependencies for AwsClientFactory

Open
#7,516 2 comments 3 reactions 0 assignees View on GitHub
not-stale
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Feature Request / Improvement

As we briefly discussed in a previous community sync, the `AwsClientFactory` class currently forces all AWS client dependencies to be required at runtime. If I just want to use `S3FileIO`, I also need to include `glue`, `dynamodb`, etc. for the whole runtime jar to work.

Many users just dump the `bundle` jar for all AWS clients, which is a huge 450MB jar and definitely not a good practice.

I was having the idea of just using separated small client factories. For example add `S3FileIOAwsClientFactory` with `s3.client-factory-impl` catalog property for `S3FileIO`. And if that config is not set, we load `AwsClientFactory` and use the clients offered there for backwards compatibility purpose.

Any thoughts? @amogh-jahagirdar @nastra @danielcweeks @singhpk234 @rajarshisarkar @akshayakp97

### Query engine

None

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.