apache / apache/iceberg-rust

Consider Using object_store as IO Abstraction

Open
#172 47 comments 6 reactions 0 assignees View on GitHub
not-stale
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

I have debated filing this ticket for a while, but largely held off as I wasn't sure how well it would be received, especially as I am acutely aware that this crate currently makes use of OpenDAL and @Xuanwo is an active contributor to both repositories. However, I feel it is important to have these discussions, and part of my role as a maintainer of object_store is to engage with others in the community and hear about how its offering could be made more compelling.

That all being said, I think [object_store](https://crates.io/crates/object_store) provides some quite compelling functionality that might be of particular interest to this project:

* First-party integration with [arrow-rs](https://docs.rs/arrow-csv/latest/arrow_csv/reader/index.html#async-usage), [parquet](https://docs.rs/parquet/50.0.0/parquet/arrow/async_reader/struct.ParquetObjectReader.html), [DataFusion](https://docs.rs/datafusion/latest/datafusion/datasource/object_store/trait.ObjectStoreRegistry.html) and [polars](https://docs.rs/polars-io/0.36.2/polars_io/cloud/fn.build_object_store.html), including sophisticated [vectored](https://docs.rs/object_store/latest/object_store/#vectored-read) and [streaming](https://docs.rs/object_store/latest/object_store/struct.GetResult.html#method.into_stream) IO
* Support for [conditional writes](https://docs.rs/object_store/latest/object_store/#conditional-put), which would allow iceberg-rs to support multiple concurrent writers directly against object storage, without needing an external catalog
* A flexible [configuration system](https://docs.rs/object_store/latest/object_store/#configuration-system) developed in partnership with, and used by both the polars and delta-rs communities
* Extensive support for the various cloud provider credential sources, with [extension points](https://docs.rs/object_store/latest/object_store/trait.CredentialProvider.html) for users to further customise this
* APIs that mirror that of object stores and not [filesystems](https://docs.rs/object_store/latest/object_store/#why-not-a-filesystem-interface), which helps to understand what and how IO is being performed, and allows support for object store specific functionality like [tags](https://docs.rs/object_store/latest/object_store/struct.PutOptions.html#structfield.tags), [partial range](https://docs.rs/object_store/latest/object_store/enum.GetRange.html) requests, and [more](https://docs.rs/object_store/latest/object_store/#conditional-fetch)...
* Battle-tested in multiple production systems, and with a substantial and [growing](https://crates.io/crates/object_store/reverse_dependencies) user-base

The major area object_store is limited, somewhat intentionally, is in the number of first-party implementations; only supporting S3-compatible stores, Google Cloud Storage, Azure Blob Storage, in-memory and local filesystems. However, the object-safe design does allow for third-party implementations, for things like [HDFS](https://github.com/datafusion-contrib/datafusion-objectstore-hdfs).

I look forward to hearing your thoughts, but also fully understand if this is not a discussion you would like to engage with at this time.

Contributor guide

Open the contributing guide

Research direction

No repository files or tests are identified in the issue. Start by reviewing the current OpenDAL-based IO integration and the linked object_store capabilities; done would require a migration scope and implementation plan agreed by maintainers.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.