apache / apache/datafusion

Make object_store support optional for embedded DataFusion

Open
#25,145 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Applications embedding DataFusion with their own `TableProvider` and `ExecutionPlan` implementations still pull in `object_store` when DataFusion's default features are disabled. These applications may provide their own file access and decoding, or use a different `object_store` version in their storage layer. They need a way to exclude DataFusion's storage dependency while retaining query planning and execution.

### Describe the solution you'd like

Add a default-enabled `object_store` feature to the existing packages. Users should be able to disable default features and select SQL and expression features without including any `object_store` package in the enabled dependency graph.

The compatibility contract should be:

- Default builds retain existing storage behavior and APIs, including direct users of child crates such as `datafusion-execution` and `datafusion-physical-plan`.
- Existing users with `default-features = false` who need storage APIs explicitly enable `object_store`. Features that require built-in storage, such as `parquet` and `avro`, enable it automatically.
- Custom providers, in-memory tables, relational query execution, memory management, and local spilling remain available without the storage feature.
- No additional DataFusion packages are required.

Acceptance should use an independent downstream consumer: `cargo tree --target all --edges all` must contain no `object_store` package when storage is disabled, and a query using a custom provider and execution plan must run successfully. Explicitly restoring storage and using default features should preserve built-in file operations. Feature unification can still re-enable storage when another dependency requests it.

### Describe alternatives you've considered

Disabling the existing default features does not remove the dependency. Aligning `object_store` versions can avoid duplicate versions but does not let an application exclude an unused storage implementation.

### Additional context

Implementation: #25144.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the existing package feature definitions and dependency graph described in the issue. Use an independent downstream consumer with `cargo tree --target all --edges all` to verify that storage-disabled builds contain no `object_store`, then run a query with a custom provider and execution plan. Done means custom-provider execution works without storage while explicitly enabled and default builds retain built-in file operations.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.