apache / apache/parquet-java

Skip the storage of repetition level and definition level for all-null column

Open
#2,022 5 comments 0 reactions 0 assignees View on GitHub
Component: Java Component: Parquet Priority: Minor Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

I have a very sparse protobuf message in my project, with thousands of fields.
In practise, most of the fields are all null values in one page.
But the repetition level and definition level takes lots of storage space.
Can parquet skip the storage of r level and d level for such all-null columns to save storage space?

**Reporter**: [Li](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=lirui-tx)

**Note**: *This issue was originally created as [PARQUET-792](https://issues.apache.org/jira/browse/PARQUET-792). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

No file, test, or entry point is named in the issue. Begin by locating the Java code that encodes repetition and definition levels, then review existing tests for all-null columns. Done means an agreed implementation and regression coverage for avoiding level storage in the described case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.