apache / apache/datafusion

`sort_fuzz` testing DX improvements

Open
#16,233 3 comments 4 reactions 0 assignees View on GitHub
development-process enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

`datafusion/core/tests/fuzz_cases` is a very cool tool! It helped to spot https://github.com/apache/datafusion/issues/16228. However, once the issue was discovered, it tool several hours to understand a problem and create a reproducible example

Partially that's because debug info to reproduce the problem right now is very big and not very useful:

Image

### Describe the solution you'd like

Ideas I have to improve the DX:
- [ ] Store dataset in a file instead of printing to stdout. The current version is too big and a bit messy - and it also makes reproducing issues harder.
- [ ] Streamline seeds. There are some hardcoded seeds, but it's unclear how to call a function with arguments to get the same error-inducing dataset. It'd help to include a hint on how to generate a test batch for debugging later.
- [ ] Add docs on how to extend, use, and maintain the suite (including the modules above).

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/core/tests/fuzz_cases and inspect the sort_fuzz testing workflow, including how datasets and seeds are currently reported. Compare the requested options for storing reproducible datasets, documenting seed or test-batch generation, and explaining how to extend and maintain the suite. Done means the selected improvements make failures easier to reproduce and the suite has the requested usage documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
developer-experience, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.