apache / apache/arrow

An Error Occured While Reading Parquet File Using C++ - GetRecordBatchReader -Corrupt snappy compressed data.

Open
#13,186 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Hi All

When I use Arrow Reading Parquet File like follow:
```
auto st = parquet::arrow::FileReader::Make(
arrow::default_memory_pool(),
parquet::ParquetFileReader::Open(_parquet, _properties), &_reader);
arrow::Status status = _reader->GetRecordBatchReader({_current_group},_parquet_column_ids, &_rb_batch);
_reader->set_batch_size(65536);
_reader->set_use_threads(true);
status = _rb_batch->ReadNext(&_batch); `
```

status is not ok and an error occured like this:
`IOError: Corrupt snappy compressed data.`

When I comment out this statement ` _reader->set_use_threads(true);`,The program runs normally and I can read parquet file well.
Program errors only occur when I read multiple columns and using `_reader->set_use_threads(true); `and a single column will not occur error

The testing parquet file is created by pyarrow,I use only 1 group and each group has 3000000 records.
The parquet file has 20 columns including int and string types

Reading file using C++,arrow 7.0.0 ,snappy 1.1.8

Writting file using python3.8 ,pyarrow 7.0.0

Looking forward to your reply

Thank you!

Contributor guide

Open the contributing guide

Research direction

Start with the parquet::arrow::FileReader::GetRecordBatchReader path and the ReadNext call shown in the report. Reproduce using the pyarrow-generated file with multiple columns and set_use_threads(true), then compare it with the single-column and non-threaded cases. Done means the threaded read no longer reports corrupt snappy data, or the failure is narrowed to a specific reader or decompression component.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.