[Discussion WIP] Allow replace of an entire partition for improved performance for reading/cleaning dataset after a delete_partition

Currently, the HUDI delete_partition will replace an entire partition by finding all file ids in each target partition, and marking them as replaced. Later on when this replacecommit instant is eligible for clean, the clean table service will find all these replaced file ids and will delete them from DFS and remove their entried from MDT partitions (like FILES). This is correct and the same with how clustering/insert_overwrite replacecommits are handled.
We have seen cases where, if we perform a delete_partition on many large partitions (large meaning 10,000+), then the subsequent clean will need more spark resources (since it has to process many file entries). For append-only datasets where we just insert files in new partitions and use delete_partition to delete old expired TTLed partitions, it would be ideal if we could avoid this cost for `clean`ing.
As an optimization, could we update HUDI such that we could mark an entire partition as "replaced"? The idea is that all relevant flows that read instants for replaced file ids would be updated to avoiding needing to read/process each individual file id entry in a replaced partition.
- The filesystem view APIs would return 0 file ids for a partition if it sees that the partition is marked as replaced in a replacecommit and no later instant has added new file ids to the partition (instead of having to update in the internal mapping of `partition -> [replace file id list]`)
- Clean planner would not attempt to find each file id in a replaced partition - it can create a smaller set of each partition that has been marked for replace and hasn't had a later instant writing new files to it. Then the clean execution can issue DFS delete call for all files in the partition. And remove all related records in MDT partitions

cc @nsivabalan

Start by tracing the filesystem view APIs and clean planner/execution paths used by delete_partition, including how replacecommit entries are read from MDT partitions. Compare handling of replaced file ids with partition-level state and identify the flows that must recognize later writes. Done means replaced partitions can be cleaned without processing every file id while preserving files added by later instants.

Written by the indexing model from the issue text.