apache / apache/datafusion-comet

Keep read concurrency in the sorted merge path

Open
#5,340 0 comments 0 reactions 1 assignee Claimed by @parthchandra View on GitHub
area:scan enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Follow up for #5323
The merge reads files more lazily than today's unordered read, which reads many files at once. For a few large files per partition this is fine, but many small files per partition could be slower.

We need to benchmark both cases, and if many-small-files regresses, then implement a fix - read ahead each file into a small bounded buffer (like Comet's repartition) so all files are read while the merge consumes them.

### Describe the potential solution

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.