apache / apache/arrow

[Parquet] pq.write_table fails with OSError: Lz4 compression failure.

Open
#50,613 0 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I encountered a table with where pq.write_table throws compression error:

```
pq.write_table(table, "foo", compression='lz4')
*** OSError: Lz4 compression failure.
```

using compression='gzip' works.

The minimal table I was able to find contains 42 rows with a single columns of strings. Each string is 30-60 mb.
The strings in the table are xml and can be compressed to ~100mb in total.

I can share the table, but would like to not do it publicly.

Python 3.12.3
pyarrow 23.0.1
lz4 4.4.5
ubuntu 24.04

### Component(s)

Parquet

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported 42-row, single-string-column case with Python's pq.write_table using compression='lz4', then compare it with compression='gzip'. The issue is resolved when the Lz4 compression failure is diagnosed and the affected write path behaves correctly for these large XML strings, with coverage for the reported case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.