Change representation of partition in FileScanConfig
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Please correct me if I am wrong, but from what I understand each partition from FileScanConfig (file_group) is executed sequentially. That means if there is large disproportion of work that needs to be done (e.g. part A 10 files 10MB, part B 10 files 10GB), then query will take as long as largest partition requires to get done.
**Describe the solution you'd like**
I would like implement work stealing by e.g. sharing emitter of PartitionedFile among FileStream's.
Possible implementations:
* Migrate FileScanConfig from { file_groups: `Vec>` } -> { file_groups: `Vec>` }, that way we keep existing interface very similar to what we have now. I would be able to make n virtual partitions that internally point to single partition.
* Alternatively migrate FileScanConfig from { file_groups: `Vec>` } -> queue/stream of files that can be shared among n workers (FileStream's, heads up naming collision)
Contributor guide
Research direction
Start by tracing FileScanConfig, file_group, PartitionedFile, and FileStream to understand how partitions are represented and executed. Compare the proposed virtual-partition and shared queue/stream approaches, then define how work stealing should preserve the existing interface while balancing uneven file groups. Done means the chosen representation supports shared work across workers and its behavior is covered by relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100