Support listing files from hive partitions with subdirectories

### Feature Request / Improvement

While migrating from hive to iceberg, i have noticed we have few subdirectories within partition folder and the migrate for those partitions didn't happen as iceberg believes there is no data since it doesn't do recursive file listing.

I see two MRs were started and closed:

https://github.com/apache/iceberg/pull/15388
https://github.com/apache/iceberg/pull/4397

Would like to work on this.

### Query engine

Spark

### Willingness to contribute

- [x] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time

Start by reviewing the issue context and the previously closed pull requests #15388 and #4397 to understand the attempted approaches to recursive listing. Trace the existing partition file discovery path from there; done means files inside subdirectories of Hive partition folders are found during migration, including the relevant coverage for Spark.

Written by the indexing model from the issue text.