apache / apache/parquet-java

Special case empty columns to store 0 pages and no column chunks in the footer

未关闭
#1,731 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: Java Component: Parquet Priority: Major Type: enhancement
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

Currently when a column is empty, each row group will contain one page that encodes repetition and definition levels for this row group. These will be as many 0s as there are rows in the row group (stored in the row group metadata). These values are encoded using RLE so it ends up being very small.
However in cases where there are a lot of columns in a very sparse dataset we end up with a lot of empty column chunks (a column chunk is the data for a given column in a given row group). The metadata could become much smaller by omitting empty column chunks as the metadata of an empty column chunk can be derived from the row count in the corresponding row group.

I propose the following:
When a column chunk is empty, do not write any page to it.
Do not add the column chunk metadata in the footer for such empty columns.
A column chunk is empty if when writing the row group to disk, there is only one page and this page contains rl and dl that are only 0s. (completely empty column).
When reading the dataset:
- the column is present in the schema.
- if there's no column chunk in the footer for a given row group that means we can just replace rls and dls with infinite streams of 0s.
- any stats information can be replaced by #rows count of nulls in predicate push down.

This will help in cases where we have huge schemas where actually a small subset of columns are populated. The file data will now look like as if we had declared only the schema for columns that actually have data in them. Only the schema in the footer will mention those empty columns.

**Reporter**: [Julien Le Dem](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=julienledem) / @julienledem

**Note**: *This issue was originally created as [PARQUET-183](https://issues.apache.org/jira/browse/PARQUET-183). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

贡献指南

这个仓库没有索引到贡献指南

调研方向

首先跟踪 issue 中描述的 row-group writer、footer 的 column-chunk 元数据处理以及 dataset reader。检查 predicate pushdown 如何使用统计信息,以及缺失的 column chunks 如何提供 repetition levels 和 definition levels。完成的标准是:空列不写入任何 page 或 footer 中的 column-chunk 元数据,同时 reader 能正确重建零 levels 和 null 统计信息。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data-engineering
Issue 类型
功能
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。