Compaction for small column files

Lance now has compact_files function to compact small fragments. The implementation now will rewrite the fragments with few rows, but this might not be sufficient enough when hundreds of columns are added. In this scenary, each column in a fragment has a small file with large row count, and will not be compacted under the current strategy. I have done some performace tests, the result is that too many small column files do affect the scan/random access performace.

I see a lance blog (Designing a Table Format for ML Workloads) mantions a strategy to deal with those files. May I ask if this function avaliable now? Or is this meaningful to compact those small column file?

Start with the existing compact_files entry point and the referenced Lance blog, then compare the current fragment-based behavior with the small-column-file scenario described here. Done means establishing whether column-file compaction is supported or should be designed, with the expected performance trade-offs and scope made explicit.