Introduce the object stores in datafusion-contrib as optional features in the datafusion core
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Currently when we utilize the datafusion core as a library, it's possible to do the remote object store registration manually in the script code. However, for a long running standalone system, like Ballista, there's no place to do the registration manually. And in general, the remote object stores supported will not be changed. Therefore, it's better to provide a mechanism to register some object stores automatically rather than manually do the registration.
**Describe the solution you'd like**
With #2065, the object store extensions will no longer depend on the datafusion core. And now the datafusion core can depend on them. We can set those object stores as optional features in the datafusion core and let users decide whether to introduce them at the compilation time.
And since those extensions are maintained in their own crates, there's not much maintenance effort for the datafusion.
**Describe alternatives you've considered**
**Additional context**
It's blocked by #2076
Contributor guide
Research direction
Start by reviewing the object store extension changes in #2065 and the blocking issue #2076, then inspect how datafusion core currently handles object store registration. The work is done when the supported object stores can be enabled as optional compilation features and are registered automatically for users who enable them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100