apache / apache/datafusion

Make `arrow-csv` and `arrow-json` optional for applicable crates

Open
#17,791 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Currently importing crates like `datafusion-expr` or `datafusion-common` pulls in `arrow` with its default features, which includes the rather heavy arrow parts `arrow-csv` and `arrow-json`. Since these parts are really only required during physical execution of the respective operators & data sources, that seems rather unnessary.

### Describe the solution you'd like

I see two options:

- set `default-features = false` for the workspace-level `arrow` dependency and then request the right features in the `Cargo.toml` files of the respective datafusion crates
- **better:** do NOT depend on the `arrow` umbrella/meta crate at all but directly on the subcrates that we need

### Describe alternatives you've considered

Keeping the dependencies simple but instead suffer from longer compile times and larger build artifacts.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the workspace-level Arrow dependency and the Cargo.toml files for datafusion-expr, datafusion-common, and the applicable DataFusion crates. Determine which crates require arrow-csv or arrow-json, then verify that unaffected crates no longer pull them in while the relevant builds and tests continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.