apache / apache/parquet-java

Limit the number of rows per block and per split

未关闭
#1,861 9 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: Java Component: Parquet Priority: Major Type: enhancement
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

We use Parquet to store raw metrics data and then query this data with Hadoop-Pig.

The issue is that sometimes we end up with small Parquet files (~80mo) that contain more than 300 000 000 rows, usually because of a constant metric which results in a very good compression. Too good. As a result we have a very few number of maps that process up to 10x more rows than the other maps and we lose the benefits of the parallelization.

The fix for that has two components I believe:
1. Be able to limit the number of rows per Parquet block (in addition to the size limit).
2. Be able to limit the number of rows per split.

**Reporter**: [Quentin Francois](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=QuentinFra)
#### Externally tracked issue: [https://github.com/apache/parquet-mr/pull/251](https://github.com/apache/parquet-mr/pull/251)

**Note**: *This issue was originally created as [PARQUET-344](https://issues.apache.org/jira/browse/PARQUET-344). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

贡献指南

这个仓库没有索引到贡献指南

调研方向

首先审查此 issue 中的 Parquet 行/块和 split 要求,以及外部跟踪的 pull request 251,重点关注 Hadoop-Pig 查询的并行化。完成标准是支持为块和 split 分别配置行数限制,同时保留现有的大小限制行为。

由索引模型根据 Issue 内容生成。

评估

技术栈
hadoop, java
领域
data-engineering, distributed-systems
Issue 类型
功能
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。