Public Access for encoder Module to Enable Generic Functionality in Parquet Crate
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 168
Description
## Is your feature request related to a problem or challenge? Please describe what you are trying to do.
In the process of implementing serialization for various data types to Parquet files using Rust, we encountered limitations due to the private visibility of the `encoder` module within the Parquet crate. Specifically, the `ColumnValueEncoder` trait, which is crucial for writing generic functions to handle a diverse set of data types efficiently, is not accessible outside of its module. This limitation prevents the creation of reusable, clean, and modular code, forcing us to duplicate logic for each data type or make less efficient implementations.
Related: [gluesql/gluesql#1269](https://github.com/gluesql/gluesql/pull/1269)
## Describe the solution you'd like
We propose making the `encoder` module public, or at least re-exporting the `ColumnValueEncoder` trait publicly, to allow developers to write generic functions for data serialization. This change would greatly enhance the usability and flexibility of the Parquet crate, enabling developers to write more abstract, reusable code that can work with any data type supported by Parquet, thereby reducing code duplication and improving the overall developer experience.
## Describe alternatives you've considered
An alternative solution would be to provide a set of predefined functions within the Parquet crate itself for handling common serialization tasks. However, this approach lacks the flexibility and extensibility offered by allowing developers to implement their own generic functions. Another alternative would be to fork the Parquet crate and make the necessary modifications, but this is not a sustainable solution as it diverges from the main codebase and requires maintaining the fork.
## Additional context
The need for this change was identified while trying to implement a comprehensive solution for writing different types of values to Parquet files. The ability to use `GenericColumnWriter` and `ColumnValueEncoder` in a generic context would significantly streamline the process of working with Parquet files in Rust, making it easier to support a wide range of data types without introducing excessive code duplication or complexity.
Contributor guide
Research direction
Start by locating the private encoder module and the ColumnValueEncoder and GenericColumnWriter definitions in the Parquet crate. Check the existing public API and related gluesql/gluesql#1269 usage; the work is done when external Rust code can use the intended encoder abstraction generically without duplicating type-specific serialization logic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100