[parquet] Consider crate to provide `object_store` crate integration
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 168
Description
### Is your feature request related to a problem or challenge?
We plan to remove explicit integration with `object_store` from the `parquet` crate, see
- https://github.com/apache/arrow-rs/pull/10354
The core rationale is to
1. Ease downstream dependency management
2. The Parquet decoder is fairly well segregated from I/O now (e.g. the PushDecoder) so it is less valuable to have a pre-existing I/O integration
3. It removes the appearance of favoritism of one I/O library over another (e.g. opendal vs object_store)
4. The amount of code is relatively small (10s of lines), so it is easy to incorporate into downstream crates and customize as needed
However it might be nicer (we think) if there was an easier way to get a pre-tested integration in a crate
### Describe the solution you'd like
The idea would be to add `ParquetObjectReader` and `ParquetObjectWriter` (that implement the object_store APIs for parquet reading / writing respectively) into a new crate (e.g. parquet-object-store)
### Describe alternatives you've considered
Some options for crate names
- `parquet-object-store` (to match https://crates.io/crates/parquet-opendal maintained by the open DAL project -- thanks @Xuanwo for the link)
### Additional context
_No response_
Contributor guide
Research direction
Start by reviewing the existing object_store integration in the parquet crate and the PushDecoder separation described in the issue. Define the proposed parquet-object-store crate around ParquetObjectReader and ParquetObjectWriter implementing the object_store APIs, with pre-tested reading and writing integration as the completion criterion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100