apache / apache/arrow

[Python] Parquet read table fail with small groups with Pyarrow 7.0.0

Open
#12,470 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

When reading a file with 7M records and groups of 10k records using Pyarrow 7.0.0, read_table crash without an error, just gave "10: SIGBUS".

If increase the size of groups to 100K it works.
If use the option use_legacy_dataset to True it works too.

Version 6.0.1 of Pyarrow works fine without the need for the use_legacy_dataset option.

```
file_name = 'output.parquet'
schema = pa.schema([
pa.field('DATE', pa.timestamp('ns')),
pa.field('OUTCOME', pa.string())
])
parquet_writer = pq.ParquetWriter(file_name,
schema,
compression='snappy',
allow_truncated_timestamps=True,
version='2.6',
data_page_version='2.0'
)

## create file with 7M records with groups size of 10K records
for i in range(700):
tbl = pa.Table.from_arrays([
pc.cast(pa.array(['2021-01-01T10:10:10'] * 10_000), pa.timestamp('ns')),
pa.array(['OK'] * 10_000),
], schema=schema)
parquet_writer.write_table(tbl)

parquet_writer.close()

### pyarrow version 7.0.0
pq.read_table(file_name, columns=['OUTCOME']) # fail 10: SIGBUS
pq.read_table(file_name, columns=['OUTCOME'], use_legacy_dataset=True) # works

### pyarrow version 6.0.1
pq.read_table(file_name, columns=['OUTCOME']) # works
pq.read_table(file_name, columns=['OUTCOME'], use_legacy_dataset=True) # works
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the provided 7M-record Parquet example with 10K-row groups under PyArrow 7.0.0, then compare default and legacy reads and the PyArrow 6.0.1 behavior. Done means reading the OUTCOME column no longer terminates with SIGBUS while preserving the existing successful cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.