Design better compaction criteria

Current issues:

1. Existing implementation only compacts fragments that are adjacent. Given that update and upsert operations re-order data anyways, it doesn't makes sense for us to limit ourselves like that.
2. We have no criteria that will trigger compaction from horizontal fragmentation. For example, if a user replaces the vector column with a new one, the dropped column will still existing alongside other metadata files. We should detect that it's worth compacting those.
3. Compaction planning doesn't differentiate the benefit of compacting small files vs compacting medium files. compaction has high ROI when many small files, but makes less sense when many medium or large files. For example, if we have 20 files each 4kb, it’s a no brainer to make those one 80kb file. But if we have 10 files each 100MB, is it actually worth rewriting as a single 1 GB file?

We should come up with a planner that is able to calculate these tradeoffs:

• What is the write amplification cost, including index remapping?
• What is the benefit to scan performance (high if many tiny files, low if files are sizeable)?
• What is the benefit to metadata overhead?

The issue names no files, tests, or entry points. Start by locating the current compaction planner and its tests, then determine how write amplification, index remapping, scan performance, and metadata overhead are represented; done means a decided planner design with criteria covering adjacent, horizontal, small-file, and larger-file compaction tradeoffs.