Difference between parquet-mr implementation and parquet-format documentation
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.6k
- 平均合并
- 3 天 12 小时
- 30 天内合并 PR
- 33
描述
Documentation at https://github.com/apache/parquet-format/blob/master/src/thrift/parquet.thrift
```
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
**/
1: optional string file_path
/** Byte offset in file_path to the ColumnMetaData **/
2: required i64 file_offset
...
```
and https://github.com/apache/parquet-format
```
4-byte magic number "PAR1"
...
```
suggests that ColumnChunk data should be followed by ColumnChunkMetaData.
However it looks like parquet-mr doesn't write ColumnMetaData after Columns at all and populates ColumnChunk.file_offset with an offset of the first data page:
from **ParquetMetadataConverter.java:153**:
```Java
for (ColumnChunkMetaData columnMetaData : columns) {
ColumnChunk columnChunk = new ColumnChunk(columnMetaData.getFirstDataPageOffset()); // verify this is the right offset
columnChunk.file_path = block.getPath(); // they are in the same file for now
```
Is it a bug in parquet-mr or in the documentation?
**Reporter**: [Konstantin Shaposhnikov](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=k.shaposhnikov@gmail.com) / @kostya-sh
#### PRs and other links:
- [parquet-format PR #56](https://github.com/apache/parquet-format/pull/56)
**Note**: *This issue was originally created as [PARQUET-291](https://issues.apache.org/jira/browse/PARQUET-291). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
贡献指南
这个仓库没有索引到贡献指南
调研方向
从 ParquetMetadataConverter.java 第 153 行开始,将其对 ColumnChunk 的构造与 parquet.thrift 中的 ColumnChunk 定义以及 parquet-format 布局文档进行比较。查看链接的 parquet-format PR #56 和 issue 中引用的迁移文档。当实现与文档之间的差异已有明确记录的解决方案时,即视为完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java
- 领域
- data-engineering
- Issue 类型
- 文档
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100