apache / apache/parquet-java

Difference between parquet-mr implementation and parquet-format documentation

未关闭
#1,819 3 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: Format Component: Java Component: Parquet Priority: Major Type: bug
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

Documentation at https://github.com/apache/parquet-format/blob/master/src/thrift/parquet.thrift

```
struct ColumnChunk {
/** File where column data is stored. If not set, assumed to be same file as
* metadata. This path is relative to the current file.
**/
1: optional string file_path

/** Byte offset in file_path to the ColumnMetaData **/
2: required i64 file_offset

...
```

and https://github.com/apache/parquet-format

```
4-byte magic number "PAR1"

...
```

suggests that ColumnChunk data should be followed by ColumnChunkMetaData.

However it looks like parquet-mr doesn't write ColumnMetaData after Columns at all and populates ColumnChunk.file_offset with an offset of the first data page:

from **ParquetMetadataConverter.java:153**:
```Java
for (ColumnChunkMetaData columnMetaData : columns) {
ColumnChunk columnChunk = new ColumnChunk(columnMetaData.getFirstDataPageOffset()); // verify this is the right offset
columnChunk.file_path = block.getPath(); // they are in the same file for now
```

Is it a bug in parquet-mr or in the documentation?

**Reporter**: [Konstantin Shaposhnikov](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=k.shaposhnikov@gmail.com) / @kostya-sh
#### PRs and other links:
- [parquet-format PR #56](https://github.com/apache/parquet-format/pull/56)

**Note**: *This issue was originally created as [PARQUET-291](https://issues.apache.org/jira/browse/PARQUET-291). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 ParquetMetadataConverter.java 第 153 行开始,将其对 ColumnChunk 的构造与 parquet.thrift 中的 ColumnChunk 定义以及 parquet-format 布局文档进行比较。查看链接的 parquet-format PR #56 和 issue 中引用的迁移文档。当实现与文档之间的差异已有明确记录的解决方案时,即视为完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data-engineering
Issue 类型
文档
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。