Make object_store support optional for embedded DataFusion
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Applications embedding DataFusion with their own `TableProvider` and `ExecutionPlan` implementations still pull in `object_store` when DataFusion's default features are disabled. These applications may provide their own file access and decoding, or use a different `object_store` version in their storage layer. They need a way to exclude DataFusion's storage dependency while retaining query planning and execution.
### Describe the solution you'd like
Add a default-enabled `object_store` feature to the existing packages. Users should be able to disable default features and select SQL and expression features without including any `object_store` package in the enabled dependency graph.
The compatibility contract should be:
- Default builds retain existing storage behavior and APIs, including direct users of child crates such as `datafusion-execution` and `datafusion-physical-plan`.
- Existing users with `default-features = false` who need storage APIs explicitly enable `object_store`. Features that require built-in storage, such as `parquet` and `avro`, enable it automatically.
- Custom providers, in-memory tables, relational query execution, memory management, and local spilling remain available without the storage feature.
- No additional DataFusion packages are required.
Acceptance should use an independent downstream consumer: `cargo tree --target all --edges all` must contain no `object_store` package when storage is disabled, and a query using a custom provider and execution plan must run successfully. Explicitly restoring storage and using default features should preserve built-in file operations. Feature unification can still re-enable storage when another dependency requests it.
### Describe alternatives you've considered
Disabling the existing default features does not remove the dependency. Aligning `object_store` versions can avoid duplicate versions but does not let an application exclude an unused storage implementation.
### Additional context
Implementation: #25144.
Contributor guide
Research direction
Start by tracing the existing package feature definitions and dependency graph described in the issue. Use an independent downstream consumer with `cargo tree --target all --edges all` to verify that storage-disabled builds contain no `object_store`, then run a query with a custom provider and execution plan. Done means custom-provider execution works without storage while explicitly enabled and default builds retain built-in file operations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100