[Python] Parquet read table fail with small groups with Pyarrow 7.0.0
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
When reading a file with 7M records and groups of 10k records using Pyarrow 7.0.0, read_table crash without an error, just gave "10: SIGBUS".
If increase the size of groups to 100K it works.
If use the option use_legacy_dataset to True it works too.
Version 6.0.1 of Pyarrow works fine without the need for the use_legacy_dataset option.
```
file_name = 'output.parquet'
schema = pa.schema([
pa.field('DATE', pa.timestamp('ns')),
pa.field('OUTCOME', pa.string())
])
parquet_writer = pq.ParquetWriter(file_name,
schema,
compression='snappy',
allow_truncated_timestamps=True,
version='2.6',
data_page_version='2.0'
)
## create file with 7M records with groups size of 10K records
for i in range(700):
tbl = pa.Table.from_arrays([
pc.cast(pa.array(['2021-01-01T10:10:10'] * 10_000), pa.timestamp('ns')),
pa.array(['OK'] * 10_000),
], schema=schema)
parquet_writer.write_table(tbl)
parquet_writer.close()
### pyarrow version 7.0.0
pq.read_table(file_name, columns=['OUTCOME']) # fail 10: SIGBUS
pq.read_table(file_name, columns=['OUTCOME'], use_legacy_dataset=True) # works
### pyarrow version 6.0.1
pq.read_table(file_name, columns=['OUTCOME']) # works
pq.read_table(file_name, columns=['OUTCOME'], use_legacy_dataset=True) # works
```
Contributor guide
Research direction
Start by reproducing the provided 7M-record Parquet example with 10K-row groups under PyArrow 7.0.0, then compare default and legacy reads and the PyArrow 6.0.1 behavior. Done means reading the OUTCOME column no longer terminates with SIGBUS while preserving the existing successful cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100