apache / apache/parquet-java

parquet-cli unable to read UUID values

未关闭
#1,641 13 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: CLI Component: Parquet Priority: Minor Type: bug
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

I am finding that parquet-cli throws when trying to read UUID values. 

Attached to this bug report is a parquet file with 2 columns, message encoded as byte-array and number encoded as fixed length byte array (UUID). This file was written by my .net implementation of parquet specification. The file has one row worth of data and is readable by parquet-cpp.

Schema as read by parquet-cli:

message root

{   required binary Message (STRING);   required fixed_len_byte_array(16) Number (UUID); }

Values as read by parquet-cpp:

— Values —
Message                       |Number                        |
First record                  |215 48 212 219 218 57 169 67 166 116 7 79 44 227 50 17 |

 

Here is the exception stack from parquet-cli when trying to read uuid values:
```java

Caused by: org.apache.parquet.io.ParquetDecodingException: The requested schema is not compatible with the file schema. incompatible types: required binary Number (STRING) != required fixed_len_byte_array(16) Number (UUID)
        at org.apache.parquet.io.ColumnIOFactory$ColumnIOCreatorVisitor.incompatibleSchema(ColumnIOFactory.java:101)
        at org.apache.parquet.io.ColumnIOFactory$ColumnIOCreatorVisitor.visit(ColumnIOFactory.java:93)
        at org.apache.parquet.schema.PrimitiveType.accept(PrimitiveType.java:602)
        at org.apache.parquet.io.ColumnIOFactory$ColumnIOCreatorVisitor.visitChildren(ColumnIOFactory.java:83)
        at org.apache.parquet.io.ColumnIOFactory$ColumnIOCreatorVisitor.visit(ColumnIOFactory.java:57)
        at org.apache.parquet.schema.MessageType.accept(MessageType.java:55)
        at org.apache.parquet.io.ColumnIOFactory.getColumnIO(ColumnIOFactory.java:162)
        at org.apache.parquet.hadoop.InternalParquetRecordReader.checkRead(InternalParquetRecordReader.java:135)
        at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:225)

```
 I debugged parquet-cli code and found that parquet-cli is trying to project the UUID as a string and later on that throws as these types are not compatible? 

 

Source code references:

At AvroReadSupport.java, line 97

~~~~~~~~~~~~

    String requestedProjectionString = configuration.get(AVRO_REQUESTED_PROJECTION);

    if (requestedProjectionString != null)

{       Schema avroRequestedProjection = new Schema.Parser().parse(requestedProjectionString);       projection = new AvroSchemaConverter(configuration).convert(avroRequestedProjection);     }

~~~~~~~~~~~~

 

Debugger values for

requestedProjectionString=

{"type":"record","name":"root","fields":[
{"name":"Message","type":"string"}

,\{"name":"Number","type":{"type":"string","logicalType":"uuid"}}]}

[Note that `Number` now has a type of `string` and a logicalType of `uuid`]

 

At ColumnIOFactory.java line 93

~~~~~~~~~~~~

incompatibleSchema(primitiveType, currentRequestedType);

~~~~~~~~~~~~

Debugger values for 

primitiveType = required fixed_len_byte_array(16) Number (UUID)

currentRequestedType = required binary Number (STRING)

 

and this will throw.

 

If I skip over the projection code in AvroReadSupport, parquet-cli is able to read my file.

I am not sure if the bug is in parquet-cli or parquet-mr or in the library I used to encode this file. The fact that parquet-cpp is able to read it gives me some confidence to say that the problem is either in parquet-cli or parquet-mr.

Please point me in the right direction if I could verify this UUID roundtripping purely from parquet-mr itself in form of an unit-test. Happy to contribute tests or fix if needed.

**Reporter**: [Balaji K](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=gamaken)
#### Original Issue Attachments:
- [guid.parquet](https://issues.apache.org/jira/secure/attachment/13042816/guid.parquet)

**Note**: *This issue was originally created as [PARQUET-2140](https://issues.apache.org/jira/browse/PARQUET-2140). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 AvroReadSupport.java 中请求的投影转换部分以及 ColumnIOFactory.java 中不兼容 schema 的检查部分开始。使用附带的 guid.parquet 文件,或使用报告中所示 UUID schema 的 parquet-mr 单元测试来复现该失败。当 parquet-cli 或相关的 parquet-mr 读取路径能够读取 fixed_len_byte_array UUID,且不再出现 schema 不兼容异常时,即视为完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data-engineering
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
冷清
描述清晰度
基本清楚
新手友好度
45/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。