apache / apache/parquet-java

Parquet-tools cat mangles strings created by other clients

Open
#2,140 0 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Priority: Major Type: bug
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

I have some parquet files that are created by Java MR process (which I do not own). I am able to read these fields successfully in pig and Spark, but for some reason the String fields are being mangled when I view the files with parquet-tools (cat).

Here are the details on the file metadata using today's build of parquet-tools:
```

hadoop jar parquet-tools-1.9.1-SNAPSHOT.jar meta /parquet-r-00000
```
Output:
```

file: hdfs:///parquet-r-00000
creator: parquet-mr version 1.8.1 (build 4aba4dae7bb0d4edbcf7923ae1339f28fd3f7fcf)

file schema: MY_DATA
--------------------------------------------------------------------------------
myfield: OPTIONAL BINARY R:0 D:1

row group 1: RC:37343 TS:32397576 OFFSET:4
--------------------------------------------------------------------------------
myfield: BINARY SNAPPY DO:0 FPO:4 SZ:273374/556406/2.04 VC:37343 ENC:RLE,BIT_PACKED,PLAIN_DICTIONARY ST:[no stats for this column]
```
 Has anyone seen this before?

**Environment**: {noformat}
uname -a
Linux myhost 4.4.0-63-generic #84-Ubuntu SMP Wed Feb 1 17:20:32 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
{noformat}
**Reporter**: [Michael McCarthy](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Michael.McCarthy)

**Note**: *This issue was originally created as [PARQUET-1235](https://issues.apache.org/jira/browse/PARQUET-1235). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the parquet-tools cat path and the metadata for the OPTIONAL BINARY field described in the report, then compare its output with the strings read by Pig and Spark. Reproduce the mangling using a Parquet file created by the Java MR process; done means parquet-tools cat displays those string fields correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
cli, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.