Parquet OutputFormat should allow controlling the file size
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.6k
- 平均合并
- 3 天 12 小时
- 30 天内合并 PR
- 33
描述
To generate the most efficient on disk file, the size of the file is important to control. It would be nice if we could configure the ouputformat to roll over new files when it reaches the right size and start a new file.
There's currently no easy way to tune this and requires indirect tuning (number of reduces, map input size).
**Reporter**: [Nong Li](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=nongli) / @nongli
**Note**: *This issue was originally created as [PARQUET-17](https://issues.apache.org/jira/browse/PARQUET-17). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
贡献指南
这个仓库没有索引到贡献指南
调研方向
该 issue 提到了 Parquet OutputFormat,但没有指定任何文件或测试。首先定位这个入口点,并确定当前如何处理输出文件大小;完成的标准是提供一个可配置的大小阈值,在达到该阈值时将输出滚动到新文件中。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java
- 领域
- data-engineering
- Issue 类型
- 功能
- 难度
- 5/5
- 预计耗时
- 一周以上
- 活跃度
- 停滞
- 描述清晰度
- 需要澄清
- 新手友好度
- 25/100