apache / apache/parquet-java

RunLengthBitPackingHybridDecoder: Reading past RLE/BitPacking stream.

Open
#1,603 6 comments 0 reactions 0 assignees View on GitHub
Component: Java Component: Parquet Priority: Major Type: bug
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

I am using Avro and Crunch 0.11 to write data into Hadoop CDH 4.6 in parquet format. This works fine for a few gigabytes but blows up in the RunLengthBitPackingHybridDecoder when reading a few thousands gigabytes.

```Java
parquet.io.ParquetDecodingException: Can not read value at 19453 in block 0 in file hdfs://nn-ix01.se-ix.delta.prod:8020/user/stoffe/parquet/dogfight/2014/09/29/part-m-00153.snappy.parquet
at parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:177)
at parquet.hadoop.ParquetRecordReader.nextKeyValue(ParquetRecordReader.java:130)
at org.apache.crunch.impl.mr.run.CrunchRecordReader.nextKeyValue(CrunchRecordReader.java:157)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:483)
at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:76)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:85)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:139)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: parquet.io.ParquetDecodingException: Can't read value in column [action] BINARY at value 697332 out of 872236, 96921 out of 96921 in currentPage. repetition level: 0, definition level: 1
at parquet.column.impl.ColumnReaderImpl.readValue(ColumnReaderImpl.java:466)
at parquet.column.impl.ColumnReaderImpl.getBinary(ColumnReaderImpl.java:414)
at parquet.filter.ColumnPredicates$1.apply(ColumnPredicates.java:64)
at parquet.filter.ColumnRecordFilter.isMatch(ColumnRecordFilter.java:69)
at parquet.io.FilteredRecordReader.skipToMatch(FilteredRecordReader.java:71)
at parquet.io.FilteredRecordReader.read(FilteredRecordReader.java:57)
at parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:173)
... 13 more
Caused by: java.lang.IllegalArgumentException: Reading past RLE/BitPacking stream.
at parquet.Preconditions.checkArgument(Preconditions.java:47)
at parquet.column.values.rle.RunLengthBitPackingHybridDecoder.readNext(RunLengthBitPackingHybridDecoder.java:80)
at parquet.column.values.rle.RunLengthBitPackingHybridDecoder.readInt(RunLengthBitPackingHybridDecoder.java:62)
at parquet.column.values.dictionary.DictionaryValuesReader.readBytes(DictionaryValuesReader.java:73)
at parquet.column.impl.ColumnReaderImpl$2$7.read(ColumnReaderImpl.java:311)
at parquet.column.impl.ColumnReaderImpl.readValue(ColumnReaderImpl.java:462)
... 19 more
```

**Environment**: Java 1.7 Linux Debian
**Reporter**: [Kristoffer Sjögren](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=deephacks) / @krisskross
**Assignee**: [Reuben Kuhnert](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=sircodesalot) / @sircodesalotOfTheRound

**Note**: *This issue was originally created as [PARQUET-112](https://issues.apache.org/jira/browse/PARQUET-112). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.