reader_writer2.cc seems to produce bad parquet file.
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the usage question you have. Please include as many useful details as possible.
Has anyone encountered the following error message when trying to build and run the "reader_writer.cc" C++ example on arrow 9?
`Parquet read error: IOError: Corrupt snappy compressed data`
Important Notes:
1. I commented out `assert(row_group_reader->metadata()->total_byte_size() < ROW_GROUP_SIZE);` because it seemed that the total bytes written were greater than the expectation. However, I wanted to see if the produced parquet file could still be read by the sample code.
2. I am forced to use g++ 5.4.0 and c++14. I built both arrow 9 (make target parquet-all) and the example with this compiler with no compiler/linker errors.
Here is the cmake command I used before building arrow 9:
cmake -B=
-DBOOST_ROOT=
-DARROW_PARQUET="ON"
-DARROW_WITH_SNAPPY="ON"
-DARROW_BUILD_STATIC="OFF"
-DARROW_WITH_RE2="OFF"
-DARROW_WITH_UTF8PROC="OFF"
-DCMAKE_BUILD_TYPE="Release"
-DCMAKE_INSTALL_PREFIX:PATH=
-DCMAKE_CXX_STANDARD="14"
-DCMAKE_CXX_COMPILER=
-DCMAKE_C_COMPILER=
### Component(s)
C++, Parquet
Contributor guide
Research direction
Start with the reader_writer.cc C++ example and the Arrow 9 Parquet/Snappy build configured by the supplied CMake command. Reproduce the write-and-read flow with g++ 5.4.0 and C++14, then inspect whether the generated Parquet file consistently triggers the reported Snappy corruption error. Done means identifying the cause or documenting the supported configuration and confirming a readable output file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100