apache / apache/datafusion

discuss: Introduce `datafusion-storage` as datafusion's own storage interface

Open
#14,854 9 comments 8 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Hello everyone, I'm jumping here from [[Discussion] Object Store Composition](https://github.com/apache/arrow-rs-object-store/issues/16).

## Background

Datafusion is using `ObjectStore` as it's public storage interface for now. We have public API like [`register_object_store`](https://docs.rs/datafusion/45.0.0/datafusion/execution/context/struct.SessionContext.html#method.register_object_store):

```rust
let object_store_url = ObjectStoreUrl::parse("file://").unwrap();
let object_store = object_store::local::LocalFileSystem::new();
let ctx = SessionContext::new();
// All files with the file:// url prefix will be read from the local file system
ctx.register_object_store(object_store_url.as_ref(), Arc::new(object_store));
```

With the growth of DF, we have to continuously add more features to `object_store`, making it increasingly difficult to compose, as described in [[Discussion] Object Store Composition](https://github.com/apache/arrow-rs-object-store/issues/16).

The latest example is [adding Extensions to object store GetOptions](https://github.com/apache/arrow-rs-object-store/issues/17) to allow passing tracing spans within the object store, as requested in [Improve use of tracing spans in query path](https://github.com/influxdata/influxdb/issues/25911).

It's easy to predict that `ObjectStore` will move further and further away from its initial position:

> Initially the ObjectStore API was relatively simple, consisting of a few methods to interact with object stores. As such many systems took this abstraction and used it as a generic IO abstraction, this is good and what the crate was designed for.

## Proposal

So I proposse to build `datafusion-storage` primarily focused on DataFusion's own needs while maintaining `datafusion-storage-object-store` and `datafusion-storage-opendal` separately. The benefit is that users can implement innovative features like `datafusion-storage-cudf` or `datafusion-storage-io_uring` without being constrained by the current I/O abstraction of object-store or OpenDAL.

If this becomes a reality, DataFusion can design the abstraction based on its own requirements without having to push everything upstream to `object_store`. This would allow them to maintain useful features such as context management and add additional requirements to the trait while letting `datafusion-storage-object-store` and `datafusion-storage-opendal` handle the extra work.

## Implematation

We can start by aliasing the `ObjectStore` trait within `datafusion-storage` first. Given sufficient migration time, we can then fine-tune the trait to better align with DF's specific needs.

Contributor guide

Open the contributing guide

Research direction

Start with the SessionContext::register_object_store entry point and the linked Object Store Composition discussion to understand the current public boundary and proposed integrations. Done would require agreement on the datafusion-storage abstraction, its object-store and OpenDAL implementations, and a migration plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.