Bound memory when a partition has many files

### What is the problem the feature request solves?

Follow up for #5323

The merge opens every file in the partition at once, and each open file allocates an Arrow read buffer. Partitions with a very large number of files can run out of memory on large datasets.

### Describe the potential solution

read a fixed number of files at a time instead of all of them, and order which files to read first by some stats (or another signal), while still producing correct merged order.

### Additional context

_No response_