apache / apache/datafusion

Make session setup an extension point for the CLI

Open
#21,972 0 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

My ultimate goal is to be able to benchmark the [Vortex](https://github.com/vortex-data/vortex)/DataFusion integration in a consist and reproducible way, especially for ClickBench.

While we have our own tooling, it tends to do through many changes and is not always a perfect match to what DataFusion does, which ends up creating unexpected differences in results.

### Describe the solution you'd like

I would like to be able to use the `datafusion-cli` crate to build our own tool, which should be as close as possible to the existing CLI, just with the ability to modify the session. I think it will be useful for us to include our `FileFormat` stuff, but I can imagine it being useful to test any other extension point in DataFusion that isn't part of the main repo.

### Describe alternatives you've considered

For the time being, I've just created https://github.com/vortex-data/vortex-datafusion-cli which should also help with our versioning, but could be more focused if there's positive response to this idea.

Contributor guide

Open the contributing guide

Research direction

Start with the datafusion-cli crate and compare its existing CLI setup with the vortex-datafusion-cli tool linked in the issue. Define what session customization should be exposed while keeping behavior close to the existing CLI; done means an external tool can modify the session, including adding FileFormat support, without maintaining a separate CLI implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.