apache / apache/parquet-java

Harden variant decoding

Open
#3,561 0 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

### Describe the enhancement requested

org.apache.parquet.variant.Variant is brittle against malformed variants, whereas the arrow variant is very strict and includes an optional deep validate phase

* Harden the java implementation, where it doesn't impose any performance penalty
* add tests which can also generate malformed files to add to the parquet test files

The work in #3481 should include some of this or this should be an immediate followon (preferred)

### Component(s)

parquet-variant

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading org.apache.parquet.variant.Variant and the parquet-variant component, then compare its malformed-input handling with the Arrow variant and review the work in #3481. Done means Java variant decoding is hardened without unnecessary performance cost, with tests that generate malformed files and add them to the Parquet test files.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.