apache / apache/datafusion

Introduce the object stores in datafusion-contrib as optional features in the datafusion core

Open
#2,072 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Currently when we utilize the datafusion core as a library, it's possible to do the remote object store registration manually in the script code. However, for a long running standalone system, like Ballista, there's no place to do the registration manually. And in general, the remote object stores supported will not be changed. Therefore, it's better to provide a mechanism to register some object stores automatically rather than manually do the registration.

**Describe the solution you'd like**

With #2065, the object store extensions will no longer depend on the datafusion core. And now the datafusion core can depend on them. We can set those object stores as optional features in the datafusion core and let users decide whether to introduce them at the compilation time.

And since those extensions are maintained in their own crates, there's not much maintenance effort for the datafusion.

**Describe alternatives you've considered**

**Additional context**

It's blocked by #2076

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the object store extension changes in #2065 and the blocking issue #2076, then inspect how datafusion core currently handles object store registration. The work is done when the supported object stores can be enabled as optional compilation features and are registered automatically for users who enable them.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.