Avoid loading all AWS client dependencies for AwsClientFactory
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Feature Request / Improvement
As we briefly discussed in a previous community sync, the `AwsClientFactory` class currently forces all AWS client dependencies to be required at runtime. If I just want to use `S3FileIO`, I also need to include `glue`, `dynamodb`, etc. for the whole runtime jar to work.
Many users just dump the `bundle` jar for all AWS clients, which is a huge 450MB jar and definitely not a good practice.
I was having the idea of just using separated small client factories. For example add `S3FileIOAwsClientFactory` with `s3.client-factory-impl` catalog property for `S3FileIO`. And if that config is not set, we load `AwsClientFactory` and use the clients offered there for backwards compatibility purpose.
Any thoughts? @amogh-jahagirdar @nastra @danielcweeks @singhpk234 @rajarshisarkar @akshayakp97
### Query engine
None
Contributor guide
Assessment
This issue has not been assessed yet.