apache / apache/datafusion

Change representation of partition in FileScanConfig

Open
#4,295 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Please correct me if I am wrong, but from what I understand each partition from FileScanConfig (file_group) is executed sequentially. That means if there is large disproportion of work that needs to be done (e.g. part A 10 files 10MB, part B 10 files 10GB), then query will take as long as largest partition requires to get done.

**Describe the solution you'd like**
I would like implement work stealing by e.g. sharing emitter of PartitionedFile among FileStream's.

Possible implementations:
* Migrate FileScanConfig from { file_groups: `Vec>` } -> { file_groups: `Vec>` }, that way we keep existing interface very similar to what we have now. I would be able to make n virtual partitions that internally point to single partition.
* Alternatively migrate FileScanConfig from { file_groups: `Vec>` } -> queue/stream of files that can be shared among n workers (FileStream's, heads up naming collision)

Contributor guide

Open the contributing guide

Research direction

Start by tracing FileScanConfig, file_group, PartitionedFile, and FileStream to understand how partitions are represented and executed. Compare the proposed virtual-partition and shared queue/stream approaches, then define how work stealing should preserve the existing interface while balancing uneven file groups. Done means the chosen representation supports shared work across workers and its behavior is covered by relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.