apache / apache/arrow-rs

[parquet] Consider crate to provide `object_store` crate integration

Open
#10,501 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

### Is your feature request related to a problem or challenge?

We plan to remove explicit integration with `object_store` from the `parquet` crate, see
- https://github.com/apache/arrow-rs/pull/10354

The core rationale is to
1. Ease downstream dependency management
2. The Parquet decoder is fairly well segregated from I/O now (e.g. the PushDecoder) so it is less valuable to have a pre-existing I/O integration
3. It removes the appearance of favoritism of one I/O library over another (e.g. opendal vs object_store)
4. The amount of code is relatively small (10s of lines), so it is easy to incorporate into downstream crates and customize as needed

However it might be nicer (we think) if there was an easier way to get a pre-tested integration in a crate

### Describe the solution you'd like

The idea would be to add `ParquetObjectReader` and `ParquetObjectWriter` (that implement the object_store APIs for parquet reading / writing respectively) into a new crate (e.g. parquet-object-store)

### Describe alternatives you've considered

Some options for crate names
- `parquet-object-store` (to match https://crates.io/crates/parquet-opendal maintained by the open DAL project -- thanks @Xuanwo for the link)

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing object_store integration in the parquet crate and the PushDecoder separation described in the issue. Define the proposed parquet-object-store crate around ParquetObjectReader and ParquetObjectWriter implementing the object_store APIs, with pre-tested reading and writing integration as the completion criterion.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.