Core: Improve HadoopFileIO performance when working with cloud storage

### Feature Request / Improvement

The class HadoopFileIO uses the hadoop 2.x APIs for opening and creating files. By moving to the newer createFile/openFile APIs and passing in information, it should be possible to

* Eliminate head requests when opening files (s3a, abfs)
* Switch the cloud connector to an optimal read policy for the known file type (random, sequential,..) (s3a, gcs)
* Have the s3a connector skip checks that creating a file won't overwrite a directory. This is consistent with S3OutputFile and relies on iceberg creating unique filenames, or at least not creating files above other objects.

There are some other minor tunings.

Constraints
1. All of this shall use APIs in Hadoop 3.3.5 so there's no need to use reflection for spark 3.4+ use.
2. There shall not be any adverse consequences when running against hdfs or local fs.

### Query engine

None

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Start with the HadoopFileIO class and the Hadoop 3.3.5 createFile/openFile APIs. Trace how files are opened and created for s3a, abfs, gcs, HDFS, and local filesystems, then verify that the changes avoid unnecessary requests and preserve HDFS and local filesystem behavior.

Written by the indexing model from the issue text.