apache / apache/datafusion-comet
Keep read concurrency in the sorted merge path
Open
area:scan
enhancement
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Follow up for #5323
The merge reads files more lazily than today's unordered read, which reads many files at once. For a few large files per partition this is fine, but many small files per partition could be slower.
We need to benchmark both cases, and if many-small-files regresses, then implement a fix - read ahead each file into a small bounded buffer (like Comet's repartition) so all files are read while the merge consumes them.
### Describe the potential solution
_No response_
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.