apache / apache/datafusion

Improve EnforceDistribution testings.

Open
#15,003 3 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

We have two primary goals: (1) ensure the test coverage includes use cases we encounter in prod, and (2) make the existing test suite easier to understand.

### Describe the solution you'd like

For the existing tests:
- [x] use an easier to read formatting of plans:
* https://github.com/apache/datafusion/pull/15007
- [x] convert the [existing macro with up to 8 vargs](https://github.com/apache/datafusion/blob/6c95b562cba95fdce90090b92657a6b464587bda/datafusion/core/tests/physical_optimizer/enforce_distribution.rs#L374-L403), into something easier to reason about (e.g. builder pattern, etc):
* https://github.com/apache/datafusion/pull/15010
- [x] enable running any series of optimizer runs over the plans, using the new testing framework:
* https://github.com/apache/datafusion/pull/15074
- [ ] move existing tests (in that file) into the new testing framework (now possible with custom list of optimizer runs)

For the improved test coverage:
- [ ] add idempotency test cases
- [ ] have test cases not using not using batch_size=1 ([existing test cases hardcode it to =1 to force roundrobin repartitioning](https://github.com/influxdata/arrow-datafusion/blob/2fcab2ef0da474ec000d7410427b9d18afb5820b/datafusion/core/tests/physical_optimizer/enforce_distribution.rs#L404-L405)).
- [ ] have test cases include data source with statistics, otherwise we [always hit this branch point](https://github.com/influxdata/arrow-datafusion/blob/2fcab2ef0da474ec000d7410427b9d18afb5820b/datafusion/physical-optimizer/src/enforce_distribution.rs#L1118), which often forces roundrobin repartitiong.
- [ ] have test cases without join key reordering, otherwise we [never hit this branch](https://github.com/influxdata/arrow-datafusion/blob/2fcab2ef0da474ec000d7410427b9d18afb5820b/datafusion/physical-optimizer/src/enforce_distribution.rs#L209-L213).
- [ ] TBD: based on codecov reports

### Describe alternatives you've considered

N/A

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with datafusion/core/tests/physical_optimizer/enforce_distribution.rs and the referenced branches in physical-optimizer/src/enforce_distribution.rs. Review the existing testing framework and optimizer runs, then identify the remaining migration and coverage cases; done means the listed idempotency, batch-size, statistics, and join-key scenarios are represented and the targeted tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.