apache / apache/arrow

[Python] Import of pyarrow.parquet and loading of non-existing file threw exception with incompatible pandas

Aperta
#46,151 6 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Component: Python Type: bug
Lingua principale
C++
Stelle
17.1k
Fork
4.3k
Merge medio
3g 20h
PR unite (30g)
94

Descrizione

### Describe the bug, including details regarding any error messages, version, and platform.

I was doing something like this:

```python
import numpy as np
import pyarrow.parquet as pq

tab = pq.read_table("non-existing.parquet")
```

This failed with:
```
tab = pq.read_table("train-00000-of-00004.parquet")
File "/home/inferencer/.local/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 1793, in read_table
dataset = ParquetDataset(
File "/home/inferencer/.local/lib/python3.10/site-packages/pyarrow/parquet/core.py", line 1291, in __init__
import pyarrow.dataset as ds
File "/home/inferencer/.local/lib/python3.10/site-packages/pyarrow/dataset.py", line 24, in
from pyarrow._dataset import ( # noqa
File "pyarrow/_dataset.pyx", line 159, in init pyarrow._dataset
File "pyarrow/_compute.pyx", line 2715, in pyarrow._compute.Expression._scalar
File "pyarrow/scalar.pxi", line 1277, in pyarrow.lib.scalar
File "pyarrow/array.pxi", line 57, in pyarrow.lib._is_array_like
File "pyarrow/pandas-shim.pxi", line 126, in pyarrow.lib._PandasAPIShim._have_pandas_internal File "pyarrow/types.pxi", line 105, in pyarrow.lib._datatype_to_pep3118
File "pyarrow/pandas-shim.pxi", line 108, in pyarrow.lib._PandasAPIShim._check_import
File "pyarrow/pandas-shim.pxi", line 50, in pyarrow.lib._PandasAPIShim._import_pandas
File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/__init__.py", line 22, in
from pandas.compat import is_numpy_dev as _is_numpy_dev
File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/compat/__init__.py", line 15, in
from pandas.compat.numpy import ( File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/compat/numpy/__init__.py", line 4, in
from pandas.util.version import Version File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/util/__init__.py", line 1, in
from pandas.util._decorators import ( # noqa:F401 File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/util/_decorators.py", line 14, in
from pandas._libs.properties import cache_readonly # noqa:F401 File "/home/inferencer/.local/lib/python3.10/site-packages/pandas/_libs/__init__.py", line 13, in
from pandas._libs.interval import Interval
File "pandas/_libs/interval.pyx", line 1, in init pandas._libs.interval
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
```

This is quite strange, because I hoped that pyarrow does not depend on pandas / pandas installed correctly. And it was strange that this threw only at `pq.read_table` call, and even before checking if file existed.

I thought that pandas depended on pyarrow and not in reverse. Is it not true? And in any case, some explicit error message with minimal required pandas version might be more clear to the user.

This was with: `pandas 1.4.3 and pyarrow-19.0.1`

After I did `python -m pip uinstall pyarrow pandas` and reinstalled again (got `pandas-2.2.3 pyarrow-19.0.1`), the problem fixed itself, and became to error-out at the non-existing file.

Thanks!

### Component(s)

Python

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Parti dal punto di ingresso Python per parquet `pyarrow/parquet/core.py` e segui `read_table` nel percorso di importazione di `ParquetDataset` e `pyarrow.dataset` mostrato nel traceback. Riproduci il problema con pandas 1.4.3 e pyarrow 19.0.1, quindi confronta con pandas 2.2.3; il lavoro è completato quando il caso di pandas incompatibile produce un errore chiaro e intenzionale oppure si evita che fallisca prima del controllo del file mancante.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
pandas, python
Ambito
data
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.