apache / apache/datafusion

Run sqllogictests (subset) through distributed execution example

Open
#23,304 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Follow-up to #23303 (prototype in #23282).

The in-process distributed execution example validates the stage/serialize/isolate contract with a handful of handwritten integration tests (joins, sorts, multi-stage aggregates). That is enough to demonstrate the model, but the coverage is only as broad as the queries someone thought to write.

### Describe the solution you'd like

Run (a subset) of the sqllogictests through the distributed execution example, so each query is planned normally, split into stages, serialized via `datafusion-proto`, executed as isolated tasks over the in-memory exchange, and its results compared against the expected output.

### Describe alternatives you've considered

- **Keep only handwritten integration tests** in the example crate: coverage stays narrow and grows only when someone remembers to add a case.

### Additional context

- Depends on #23303; working prototype of the execution harness in #23282.
- Prior art: [datafusion-distributed](https://github.com/datafusion-contrib/datafusion-distributed) and Ballista both maintain their own differential testing against DataFusion; running the in-tree sqllogictests through an in-tree harness shortens that feedback loop to a single repo.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing #23303 and the prototype in #23282, then inspect the in-process distributed execution example, its handwritten integration tests, and the in-tree sqllogictests harness. Done means a subset of sqllogictests runs through normal planning, stage splitting, datafusion-proto serialization, isolated tasks, and the in-memory exchange, with results compared to expected output.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
distributed-systems, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.