apache / apache/datafusion

Support reduced dependency graph size for datafusion-cli

Open
#23,412 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

The `datafusion-cli` crate currently pulls in S3, GCS and generic HTTP support unconditionally. This functionality pulls in a number of other crates whose transitive closure of dependencies is quite large. When tracking dependency vulnerabilities for all of these, this can lead to quite a lot of false positives when the cloud functionality is not actually used. Additionally, this increases binary size with effectively dead code.

### Describe the solution you'd like

Add features (enabled by default for backward compatibility) that can be used to omit the cloud functionality and associated dependencies.

### Describe alternatives you've considered

- Build your own CLI from scratch to have precise control over dependencies.
- Modularise the CLI into smaller library parts to make building your own easier.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in the datafusion-cli crate's dependency configuration and trace the unconditional S3, GCS, and generic HTTP support. Add default-enabled features that allow the cloud functionality and its dependencies to be omitted, then verify that default builds remain compatible and that a cloud-free build has a reduced dependency graph.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system, cli, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.