Run sqllogictests (subset) through distributed execution example
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Follow-up to #23303 (prototype in #23282).
The in-process distributed execution example validates the stage/serialize/isolate contract with a handful of handwritten integration tests (joins, sorts, multi-stage aggregates). That is enough to demonstrate the model, but the coverage is only as broad as the queries someone thought to write.
### Describe the solution you'd like
Run (a subset) of the sqllogictests through the distributed execution example, so each query is planned normally, split into stages, serialized via `datafusion-proto`, executed as isolated tasks over the in-memory exchange, and its results compared against the expected output.
### Describe alternatives you've considered
- **Keep only handwritten integration tests** in the example crate: coverage stays narrow and grows only when someone remembers to add a case.
### Additional context
- Depends on #23303; working prototype of the execution harness in #23282.
- Prior art: [datafusion-distributed](https://github.com/datafusion-contrib/datafusion-distributed) and Ballista both maintain their own differential testing against DataFusion; running the in-tree sqllogictests through an in-tree harness shortens that feedback loop to a single repo.
Contributor guide
Research direction
Start by reviewing #23303 and the prototype in #23282, then inspect the in-process distributed execution example, its handwritten integration tests, and the in-tree sqllogictests harness. Done means a subset of sqllogictests runs through normal planning, stage splitting, datafusion-proto serialization, isolated tasks, and the in-memory exchange, with results compared to expected output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- distributed-systems, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100