datafusion-contrib / datafusion-contrib/datafusion-distributed
tests and benchmarks on range partitioned data
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 139
- Forks
- 67
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 35
Description
It's not valid in datafusion to claim your data source is distributed by hash(expr), so this test isn't useful. We should migrate it to a distribution which maps to range partitioning.
There's several open questions that can only really be solved with tests
file_scan_config_scale_up_leaf_nodeinsrc/events/defaults/file_scan_config.rsif the file scan is claiming range partitioning and we rearrange files, that can cause correctness issues right? Should we callFileScanConfig::repartitioned- Say target_partitions is
10and our data source has100range partitions. Does this distribute correctly to 10 leaf tasks? Does datafusion introduce range partitions? - Are all instances of
scale_partitioningcorrect when it comes to range partitioned data? https://github.com/datafusion-contrib/datafusion-distributed/blob/a6cff424261623dbdd1e2b5cbe83e5273ebc22de/src/execution_plans/common.rs?plain=1#L17
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the range-related test in tests/join.rs, then read file_scan_config_scale_up_leaf_node in src/events/defaults/file_scan_config.rs and scale_partitioning in src/execution_plans/common.rs. Use the existing questions to determine how range partitioning behaves when files or partitions are scaled, and cover those cases with tests and benchmarks. Done means the tests exercise range-partitioned data and establish whether repartitioning and scaling are correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems, testing-qa
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100