Improve ability for users to create their own writers using the low level parquet APIs.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 168
Description
**Is your feature request related to a ~problem or~ challenge? Please describe what you are trying to do.**
We have been using at least two parquet writers that both utilize the low-level APIs provided by the parquet crate (e.g. [SerializedFileWriter](https://docs.rs/parquet/52.2.0/parquet/file/writer/struct.SerializedFileWriter.html)). One of the writers (ArrowWriter) is provided as part of the parquet crate, whereas the other parallel writer (datafusion's ParquetSink) is not. However, in both cases we later attempt to read these files using the parquet crate's readers.
The challenge is that we keep encountering unexpected differences in the parquet written by these two writers. The most recent example is that the [arrow schema is missing when using datafusion's parallel writer](https://github.com/apache/datafusion/issues/11770), whereas it is [included in the ArrowWriter](ttps://github.com/apache/arrow-rs/blob/2905ce6796cad396241fc50164970dbf1237440a/parquet/src/arrow/arrow_writer/mod.rs#L188-L190) on parquet write.
**Describe the solution you'd like**
Can we update the lower level APIs (in the parquet crate) to make it easier for users to create their own parquet writers -- without encountering surprise differences from the behavior of parquet's ArrowWriter? Provide better documentation? Provide guidance for testing when creating your own parquet writer?
**Describe alternatives you've considered**
Alternatively, we could consider this problem as the responsibility for the users that create their own writers. We already plan to file a datafusion ticket proposing that we need integration tests to ensure byte equivalency in the output parquet (vs parquet written by ArrowWriter).
**Additional context**
This is not the first time that we have discovered differences in the output parquet from the ArrowWriter vs datafusion's ParquetSink. However, we are unclear on the best way to divide responsibilities (for more testing) vs API design.
Contributor guide
Research direction
Start with the low-level SerializedFileWriter API and the ArrowWriter implementation in parquet/src/arrow/arrow_writer/mod.rs around lines 188-190, then compare the behavior with DataFusion's ParquetSink. Review the proposed integration tests for byte equivalency and Arrow schema preservation. Done should be defined as an agreed API, documentation, or testing change that prevents or clearly identifies these writer differences.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100