Parquet-tools cat mangles strings created by other clients
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
I have some parquet files that are created by Java MR process (which I do not own). I am able to read these fields successfully in pig and Spark, but for some reason the String fields are being mangled when I view the files with parquet-tools (cat).
Here are the details on the file metadata using today's build of parquet-tools:
```
hadoop jar parquet-tools-1.9.1-SNAPSHOT.jar meta /parquet-r-00000
```
Output:
```
file: hdfs:///parquet-r-00000
creator: parquet-mr version 1.8.1 (build 4aba4dae7bb0d4edbcf7923ae1339f28fd3f7fcf)
file schema: MY_DATA
--------------------------------------------------------------------------------
myfield: OPTIONAL BINARY R:0 D:1
row group 1: RC:37343 TS:32397576 OFFSET:4
--------------------------------------------------------------------------------
myfield: BINARY SNAPPY DO:0 FPO:4 SZ:273374/556406/2.04 VC:37343 ENC:RLE,BIT_PACKED,PLAIN_DICTIONARY ST:[no stats for this column]
```
Has anyone seen this before?
**Environment**: {noformat}
uname -a
Linux myhost 4.4.0-63-generic #84-Ubuntu SMP Wed Feb 1 17:20:32 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
{noformat}
**Reporter**: [Michael McCarthy](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Michael.McCarthy)
**Note**: *This issue was originally created as [PARQUET-1235](https://issues.apache.org/jira/browse/PARQUET-1235). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the parquet-tools cat path and the metadata for the OPTIONAL BINARY field described in the report, then compare its output with the strings read by Pig and Spark. Reproduce the mangling using a Parquet file created by the Java MR process; done means parquet-tools cat displays those string fields correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- cli, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100