[BUG](python): The order of the table was changed after executing the `compact_files` operation

Lance Version

pylance 0.20.0

What happened

The order of the dataset was changed after executing the compact_files operation.

My code is as follows:

import lance
from lance.dataset import DatasetOptimizer

DB_PATH = "<path-to-my-dataset>"

def main():
    dataset = lance.dataset(DB_PATH)
    print(dataset.take([0,1,2,3,4,5,6,7,8,9])  # show the first ten elements

    # compact the dataset
    optim = DatasetOptimizer(dataset)
    optim.compact_files(num_threads=8)
    print(dataset.take([0,1,2,3,4,5,6,7,8,9])  # the first ten elements was changed
    return

if __name__ == "__main__":
    main()

My enviroment

I am using a ubuntu server with 64 cores and 512G memory.
The dataset has 5 columns: title(str), section(str), text(str), id(str), and vector(list[float]).

How to reproduce

This dataset has 38 Million records of 768 dim vector and payload. I'm not sure if its feasible to share the dataset.

Start with the Python DatasetOptimizer.compact_files operation and compare dataset.take results before and after compaction. Investigate how the operation represents record order, then reproduce the reported change with a smaller dataset if possible. Done means the ordering behavior is understood and a regression test covers the intended result.