Bounded Compaction Planner To Limit the amount of data processed during a single compaction

Follow up work for https://github.com/lance-format/lance/issues/5186

During the Compaction process, it is necessary to traverse and calculate all fragments, determine which ones are suitable for Compaction, and split them into Compaction Tasks. However, in the scenario of large datasets, the following bottlenecks exist:

1. The Compaction Plan itself takes a long time to generate. In a scenario we encountered with 300,000 fragments, it may takes 13min to generate plan or even led to an OOM error.
2. A large number of Compaction Tasks are generated in the planning. Even in the case of concurrent execution, it still takes a long time to complete. Moreover, if a Task fails due to certain reasons (such as spark driver loss), the entire Compaction needs to be retried completely, which is a huge waste of resources.

In response to this, we can extend the existing design of the Compaction Planner to create a Planner that limits the computational load of Compaction (such as the total data throughput 100TB total？, the total number of fragments involved in the calculation, etc.), named BoundedCompactionPlanner. Its purpose is to reasonably split a large Compaction job into multiple sub-jobs and complete them smoothly.

Start by reading the existing Compaction Planner design and the follow-up issue #5186. Determine how a bounded planner should limit data or fragment processing and split a large compaction into sub-jobs; done means large datasets can be planned and retried in bounded portions without processing the entire job at once.