apache / apache/datafusion

Enable sort query fuzzing with limited memory

Open
#15,517 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

A new sort query fuzzer for out-of-core sorting has been added in https://github.com/apache/datafusion/pull/15501.
However, the memory limit is not enabled because there are known issues that cause external sort queries to fail, resulting in the fuzzer consistently failing. Once these issues are resolved, we can enable the memory limit configuration in the fuzzer.

After this ticket is resolved: https://github.com/apache/datafusion/issues/14748
The following update can be made to the sort query fuzzer to increase coverage for memory-limited sorts.

- [ ] Support generating queries with arbitrary number of ORDER BY clauses (Currently it's limited to be smaller than number of projected columns)
- [ ] Enable special type columns like `utf8_low` to be used in ORDER BY clauses
- [ ] Enable memory limiting functionality in the fuzzer runner
- [ ] Increase the fuzzing throughput: now it spent a lot of time verifying the results (by formatting all the sorted results), we can make it more efficient by splitting the test into smaller tests to run in parallel.

### Describe the solution you'd like

_No response_

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the sort query fuzzer introduced in pull request #15501 and the blocking work tracked in issue #14748. Done means updating the fuzzer to support the listed ORDER BY and special-type cases, enabling memory limiting, and improving verification throughput.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
backend, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
40/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.