Parquet File is not able to read from Spark and Hive
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
Issue: Caused by: java.io.IOException: Expected 35393 values in column chunk at maprfs:////path/date=20190605/caa63aa9-abfa-46e1-8221-10f6c669512d.parquet offset 4 but got 46402 values instead over 2 pages ending at file offset 341624
we are getting Avro Serialized messages from kafka which are being consumed by Spring-kafka and converted into parquet gets persisted into MaprFS(hdfs) file system.
i have tried replicating the issue in local with same Avro file but i was able to read parquet successfully, I am not sure why the parquet being corrupted in HDFS .
**Reporter**: [Prashanth pampanna desai](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Prashanth+Desai)
**Note**: *This issue was originally created as [PARQUET-1594](https://issues.apache.org/jira/browse/PARQUET-1594). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the reported IOException and compare the local successful read with the MapRFS/HDFS path used after Spring-kafka consumes Avro messages. The issue mentions no source file or test, and completion is not defined beyond determining why Spark and Hive cannot read the persisted Parquet file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, kafka, spark, spring
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100