Enable sort query fuzzing with limited memory
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
A new sort query fuzzer for out-of-core sorting has been added in https://github.com/apache/datafusion/pull/15501.
However, the memory limit is not enabled because there are known issues that cause external sort queries to fail, resulting in the fuzzer consistently failing. Once these issues are resolved, we can enable the memory limit configuration in the fuzzer.
After this ticket is resolved: https://github.com/apache/datafusion/issues/14748
The following update can be made to the sort query fuzzer to increase coverage for memory-limited sorts.
- [ ] Support generating queries with arbitrary number of ORDER BY clauses (Currently it's limited to be smaller than number of projected columns)
- [ ] Enable special type columns like `utf8_low` to be used in ORDER BY clauses
- [ ] Enable memory limiting functionality in the fuzzer runner
- [ ] Increase the fuzzing throughput: now it spent a lot of time verifying the results (by formatting all the sorted results), we can make it more efficient by splitting the test into smaller tests to run in parallel.
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reviewing the sort query fuzzer introduced in pull request #15501 and the blocking work tracked in issue #14748. Done means updating the fuzzer to support the listed ORDER BY and special-type cases, enabling memory limiting, and improving verification throughput.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- backend, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 40/100