apache / apache/arrow

reader_writer2.cc seems to produce bad parquet file.

Open
#35,897 3 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the usage question you have. Please include as many useful details as possible.

Has anyone encountered the following error message when trying to build and run the "reader_writer.cc" C++ example on arrow 9?

`Parquet read error: IOError: Corrupt snappy compressed data`

Important Notes:
1. I commented out `assert(row_group_reader->metadata()->total_byte_size() < ROW_GROUP_SIZE);` because it seemed that the total bytes written were greater than the expectation. However, I wanted to see if the produced parquet file could still be read by the sample code.

2. I am forced to use g++ 5.4.0 and c++14. I built both arrow 9 (make target parquet-all) and the example with this compiler with no compiler/linker errors.
Here is the cmake command I used before building arrow 9:

cmake -B=
-DBOOST_ROOT=
-DARROW_PARQUET="ON"
-DARROW_WITH_SNAPPY="ON"
-DARROW_BUILD_STATIC="OFF"
-DARROW_WITH_RE2="OFF"
-DARROW_WITH_UTF8PROC="OFF"
-DCMAKE_BUILD_TYPE="Release"
-DCMAKE_INSTALL_PREFIX:PATH=
-DCMAKE_CXX_STANDARD="14"
-DCMAKE_CXX_COMPILER=
-DCMAKE_C_COMPILER=

### Component(s)

C++, Parquet

Contributor guide

Open the contributing guide

Research direction

Start with the reader_writer.cc C++ example and the Arrow 9 Parquet/Snappy build configured by the supplied CMake command. Reproduce the write-and-read flow with g++ 5.4.0 and C++14, then inspect whether the generated Parquet file consistently triggers the reported Snappy corruption error. Done means identifying the cause or documenting the supported configuration and confirming a readable output file.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.