Improve EnforceDistribution testings.
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
We have two primary goals: (1) ensure the test coverage includes use cases we encounter in prod, and (2) make the existing test suite easier to understand.
### Describe the solution you'd like
For the existing tests:
- [x] use an easier to read formatting of plans:
* https://github.com/apache/datafusion/pull/15007
- [x] convert the [existing macro with up to 8 vargs](https://github.com/apache/datafusion/blob/6c95b562cba95fdce90090b92657a6b464587bda/datafusion/core/tests/physical_optimizer/enforce_distribution.rs#L374-L403), into something easier to reason about (e.g. builder pattern, etc):
* https://github.com/apache/datafusion/pull/15010
- [x] enable running any series of optimizer runs over the plans, using the new testing framework:
* https://github.com/apache/datafusion/pull/15074
- [ ] move existing tests (in that file) into the new testing framework (now possible with custom list of optimizer runs)
For the improved test coverage:
- [ ] add idempotency test cases
- [ ] have test cases not using not using batch_size=1 ([existing test cases hardcode it to =1 to force roundrobin repartitioning](https://github.com/influxdata/arrow-datafusion/blob/2fcab2ef0da474ec000d7410427b9d18afb5820b/datafusion/core/tests/physical_optimizer/enforce_distribution.rs#L404-L405)).
- [ ] have test cases include data source with statistics, otherwise we [always hit this branch point](https://github.com/influxdata/arrow-datafusion/blob/2fcab2ef0da474ec000d7410427b9d18afb5820b/datafusion/physical-optimizer/src/enforce_distribution.rs#L1118), which often forces roundrobin repartitiong.
- [ ] have test cases without join key reordering, otherwise we [never hit this branch](https://github.com/influxdata/arrow-datafusion/blob/2fcab2ef0da474ec000d7410427b9d18afb5820b/datafusion/physical-optimizer/src/enforce_distribution.rs#L209-L213).
- [ ] TBD: based on codecov reports
### Describe alternatives you've considered
N/A
### Additional context
_No response_
Contributor guide
Research direction
Start with datafusion/core/tests/physical_optimizer/enforce_distribution.rs and the referenced branches in physical-optimizer/src/enforce_distribution.rs. Review the existing testing framework and optimizer runs, then identify the remaining migration and coverage cases; done means the listed idempotency, batch-size, statistics, and join-key scenarios are represented and the targeted tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100