apache / apache/parquet-java

Introduce an interface for controlling the encoding per column

Open
#1,766 2 comments 0 reactions 0 assignees View on GitHub
Component: Java Component: Parquet Priority: Major Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

There are use-cases in which it is advantageous to have control over the type of encoding used for a given column. To achieve this, the ParquetProperties class must be refactored to facilitate its subclassing. The constructors of ParquetWriter and (probably) InternalParquetRecordWriter have to be modified as well. All the internal parquet classes have to refrain from directly instantiating ParquetProperties and should accept the instance from outside.

Currently me and my team are using a good amount of reflection and unenforceable assumptions in order to gain control of this behavior. I'd like to implement the interface change. Are you interested in this feature?

**Reporter**: [Boris Peltekov](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=peltekster) / @peltekster

**Note**: *This issue was originally created as [PARQUET-226](https://issues.apache.org/jira/browse/PARQUET-226). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading ParquetProperties and the constructors of ParquetWriter and InternalParquetRecordWriter. Trace internal Parquet classes that instantiate ParquetProperties, then define the interface and dependency flow needed to control encoding per column; done means callers can supply the encoding behavior without reflection or direct internal instantiation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.