apache / apache/arrow-rs

Public Access for encoder Module to Enable Generic Functionality in Parquet Crate

Open
#5,410 9 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

## Is your feature request related to a problem or challenge? Please describe what you are trying to do.

In the process of implementing serialization for various data types to Parquet files using Rust, we encountered limitations due to the private visibility of the `encoder` module within the Parquet crate. Specifically, the `ColumnValueEncoder` trait, which is crucial for writing generic functions to handle a diverse set of data types efficiently, is not accessible outside of its module. This limitation prevents the creation of reusable, clean, and modular code, forcing us to duplicate logic for each data type or make less efficient implementations.

Related: [gluesql/gluesql#1269](https://github.com/gluesql/gluesql/pull/1269)

## Describe the solution you'd like

We propose making the `encoder` module public, or at least re-exporting the `ColumnValueEncoder` trait publicly, to allow developers to write generic functions for data serialization. This change would greatly enhance the usability and flexibility of the Parquet crate, enabling developers to write more abstract, reusable code that can work with any data type supported by Parquet, thereby reducing code duplication and improving the overall developer experience.

## Describe alternatives you've considered

An alternative solution would be to provide a set of predefined functions within the Parquet crate itself for handling common serialization tasks. However, this approach lacks the flexibility and extensibility offered by allowing developers to implement their own generic functions. Another alternative would be to fork the Parquet crate and make the necessary modifications, but this is not a sustainable solution as it diverges from the main codebase and requires maintaining the fork.

## Additional context

The need for this change was identified while trying to implement a comprehensive solution for writing different types of values to Parquet files. The ability to use `GenericColumnWriter` and `ColumnValueEncoder` in a generic context would significantly streamline the process of working with Parquet files in Rust, making it easier to support a wide range of data types without introducing excessive code duplication or complexity.

Contributor guide

Open the contributing guide

Research direction

Start by locating the private encoder module and the ColumnValueEncoder and GenericColumnWriter definitions in the Parquet crate. Check the existing public API and related gluesql/gluesql#1269 usage; the work is done when external Rust code can use the intended encoder abstraction generically without duplicating type-specific serialization logic.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.