apache / apache/arrow

[C++][Python] Significant performance degradation after pyarrow 24.0.0 to 25.0.0 upgrade

Open
#50,667 9 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

In our application we implement a local TimeSeries caching implementation that stores TimeSeries data into Parquet files. We heavily read/write parquet files via the pandas to_parquet(engine="pyarrow") and read_parquet. We also use the pyarrow.read_metadata directly from the pyarrows package. We use default (snappy) pandas compression when we write out our files, which consist of a single column DataFrame with the index being the timestamps (DateTimeIndex) and the single column representing the TimeSeries values (mostly Float32/64).

After installing pyarrow 25.0.0 we saw a significant reduction in performance (see trend below). When we downgraded back to 24.0.0 the problem went away. No other changes were made and we kept the rest of our dependencies the same.

Image

The Dev/Prod stack Azure App Service on `Linux 779de6246efc 6.6.139.1-1.azl3`:
```
root@779de6246efc:/home/site/wwwroot# python
Python 3.13.13 (main, May 4 2026, 08:38:58) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow as pa
>>> pa.show_info()
pyarrow version info
--------------------
Package kind : python-wheel-manylinux228
Arrow C++ library version : 24.0.0
Arrow C++ compiler : GNU 14.2.1
Arrow C++ compiler flags : -Wno-noexcept-type -Wno-self-move -Wno-subobject-linkage -fdiagnostics-color=always -Wall -fno-semantic-interposition -msse4.2
Arrow C++ git revision :
Arrow C++ git description :
Arrow C++ build type : release
PyArrow build type : release

Platform:
OS / Arch : Linux x86_64
SIMD Level : avx2
Detected SIMD Level : avx2

Memory:
Default backend : mimalloc
Bytes allocated : 0 bytes
Max memory : 0 bytes
Supported Backends : mimalloc, jemalloc, system

Optional modules:
csv : Enabled
cuda : -
dataset : Enabled
feather : Enabled
flight : Enabled
fs : Enabled
gandiva : -
json : Enabled
orc : Enabled
parquet : Enabled
opentelemetry : Enabled

Filesystems:
AzureFileSystem : Enabled
GcsFileSystem : Enabled
HadoopFileSystem : Enabled
S3FileSystem : Enabled

Compression Codecs:
brotli : Enabled
bz2 : Enabled
gzip : Enabled
lz4_frame : Enabled
lz4 : Enabled
snappy : Enabled
zstd : Enabled
```

### Component(s)

C++, Python

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported workload using pandas to_parquet/read_parquet with the pyarrow engine and direct pyarrow.read_metadata calls, comparing pyarrow 24.0.0 and 25.0.0 on the described Linux environment. Measure read and write performance for the single-column timestamp-indexed Parquet files with default Snappy compression, then identify the regression and confirm that the affected operations return to the earlier performance.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, pandas, python
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.