Improve integration of hadoop s3a and comet

### What is the problem the feature request solves?

https://github.com/apache/datafusion-comet/pull/1817 introduces integration that allows usage of `hadoop-aws`'s `s3a` configuration with the `object_store` used by the native scan implementations.

Some more work can be done to improve this -

1. The feature works with `native_datafusion` but not with `native_iceberg_compat`. To be truly useful we need the same integration in `native_iceberg_compat`.

2. To allow usage of custom authentication, we need a JNI based adapter that can get credentials from an AWS credentials provider (Both AWS SDK V1 and V2 credentials providers must be supported).

3. Document usage.

### Describe the potential solution

_No response_

### Additional context

_No response_