[DISCUSSION] Make it easier and faster to query remote files (S3, iceberg, etc)

### Is your feature request related to a problem or challenge?

I personally think making it easy to use DataFusion with the "open data lake" stack is very important over the next few months.

@julienledem wrote up a very nice piece describing [The advent of the Open Data Lake](https://sympathetic.ink/2024/11/07/The-Advent-Of-The-Open-Data-Lake.html)

The high level idea is to make it really easy for people to build systems that query (quickly!) from parquet files stored on remote object store, including Apache Iceberg, Delta Lake, Hudi, etc.

You *can* already use DataFusion (and `datafusion-cli`) to query such data, but it takes non trivial effort to configure and tune for good performance. My idea is to make it easier to do so / make DataFusion better out of the box.

With that as a building block, people could/would build applications and systems targeting specific usecases

I don't yet fully understand where we currently stand on this goal, but I wanted to start hte discussio

### Describe the solution you'd like

In my mind, the specific work this entails stuff like

- [ ] Making it easier to use iceberg/delta/hudi with DataFusion
- [x] https://github.com/apache/datafusion/issues/12393
- [ ] Make parquet reader in arrow-rs faster/better on remote object stores
- [ ] Making it eaiser to cache parquet metadata
- [x] https://github.com/apache/datafusion/issues/16299
- [x] https://github.com/apache/datafusion/issues/16302
- [ ] https://github.com/apache/datafusion/issues/16303
- [x] https://github.com/apache/datafusion/issues/16306
- [x] https://github.com/apache/datafusion/issues/16460

### Describe alternatives you've considered

One specific item, brought up by @MrPowers would be to try DataFusion with the "10B row challenge" described in https://dataengineeringcentral.substack.com/p/10-billion-row-challenge-duckdb-vs .

I suspect it would be non ideal at first, but trying it to figure out what the challenges are would help us focus our efforts

### Additional context

_No response_

No implementation file or test is named. Start by reading the linked issues 12393, 16299, 16302, 16303, 16306, and 16460, then review the 10-billion-row challenge; done would require narrowing this discussion to a scoped change with measurable remote-file query performance goals.

Written by the indexing model from the issue text.