Consider Using object_store as IO Abstraction
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
I have debated filing this ticket for a while, but largely held off as I wasn't sure how well it would be received, especially as I am acutely aware that this crate currently makes use of OpenDAL and @Xuanwo is an active contributor to both repositories. However, I feel it is important to have these discussions, and part of my role as a maintainer of object_store is to engage with others in the community and hear about how its offering could be made more compelling.
That all being said, I think [object_store](https://crates.io/crates/object_store) provides some quite compelling functionality that might be of particular interest to this project:
* First-party integration with [arrow-rs](https://docs.rs/arrow-csv/latest/arrow_csv/reader/index.html#async-usage), [parquet](https://docs.rs/parquet/50.0.0/parquet/arrow/async_reader/struct.ParquetObjectReader.html), [DataFusion](https://docs.rs/datafusion/latest/datafusion/datasource/object_store/trait.ObjectStoreRegistry.html) and [polars](https://docs.rs/polars-io/0.36.2/polars_io/cloud/fn.build_object_store.html), including sophisticated [vectored](https://docs.rs/object_store/latest/object_store/#vectored-read) and [streaming](https://docs.rs/object_store/latest/object_store/struct.GetResult.html#method.into_stream) IO
* Support for [conditional writes](https://docs.rs/object_store/latest/object_store/#conditional-put), which would allow iceberg-rs to support multiple concurrent writers directly against object storage, without needing an external catalog
* A flexible [configuration system](https://docs.rs/object_store/latest/object_store/#configuration-system) developed in partnership with, and used by both the polars and delta-rs communities
* Extensive support for the various cloud provider credential sources, with [extension points](https://docs.rs/object_store/latest/object_store/trait.CredentialProvider.html) for users to further customise this
* APIs that mirror that of object stores and not [filesystems](https://docs.rs/object_store/latest/object_store/#why-not-a-filesystem-interface), which helps to understand what and how IO is being performed, and allows support for object store specific functionality like [tags](https://docs.rs/object_store/latest/object_store/struct.PutOptions.html#structfield.tags), [partial range](https://docs.rs/object_store/latest/object_store/enum.GetRange.html) requests, and [more](https://docs.rs/object_store/latest/object_store/#conditional-fetch)...
* Battle-tested in multiple production systems, and with a substantial and [growing](https://crates.io/crates/object_store/reverse_dependencies) user-base
The major area object_store is limited, somewhat intentionally, is in the number of first-party implementations; only supporting S3-compatible stores, Google Cloud Storage, Azure Blob Storage, in-memory and local filesystems. However, the object-safe design does allow for third-party implementations, for things like [HDFS](https://github.com/datafusion-contrib/datafusion-objectstore-hdfs).
I look forward to hearing your thoughts, but also fully understand if this is not a discussion you would like to engage with at this time.
Contributor guide
Research direction
No repository files or tests are identified in the issue. Start by reviewing the current OpenDAL-based IO integration and the linked object_store capabilities; done would require a migration scope and implementation plan agreed by maintainers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100