Make session setup an extension point for the CLI
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
My ultimate goal is to be able to benchmark the [Vortex](https://github.com/vortex-data/vortex)/DataFusion integration in a consist and reproducible way, especially for ClickBench.
While we have our own tooling, it tends to do through many changes and is not always a perfect match to what DataFusion does, which ends up creating unexpected differences in results.
### Describe the solution you'd like
I would like to be able to use the `datafusion-cli` crate to build our own tool, which should be as close as possible to the existing CLI, just with the ability to modify the session. I think it will be useful for us to include our `FileFormat` stuff, but I can imagine it being useful to test any other extension point in DataFusion that isn't part of the main repo.
### Describe alternatives you've considered
For the time being, I've just created https://github.com/vortex-data/vortex-datafusion-cli which should also help with our versioning, but could be more focused if there's positive response to this idea.
Contributor guide
Research direction
Start with the datafusion-cli crate and compare its existing CLI setup with the vortex-datafusion-cli tool linked in the issue. Define what session customization should be exposed while keeping behavior close to the existing CLI; done means an external tool can modify the session, including adding FileFormat support, without maintaining a separate CLI implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100