[Python] Update pandas->pyarrow conversion for the .values deprecation in pandas
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Pandas has deprecated certain aspects of `.values` (when it looses information), which our pandas->pyarrow conversion currently relies upon, so in the tests with pandas nightly we are seeing a lot of warnings because of that (https://github.com/ursacomputing/crossbow/actions/runs/34548523730/job/103106277981).
Small illustration:
```python
>>> import pandas as pd
>>> import pyarrow as pa
>>> ser = pd.Series(pd.date_range("2025-01-01", periods=3, tz="UTC"))
>>> pa.array(ser)
:1: Pandas4Warning: Series.values returning an ndarray that drops timezone information for DatetimeTZDtype is deprecated. In a future version, this will return the underlying DatetimeArray instead. Use 'Series.to_numpy()' to get a NumPy array, or 'Series.array' to get the ExtensionArray.
[
2025-01-01 00:00:00.000000Z,
2025-01-02 00:00:00.000000Z,
2025-01-03 00:00:00.000000Z
]
```
This is caused by our handling of a Series object in `pa.array(..)`:
https://github.com/apache/arrow/blob/07be48c5402adedb68c510531939a907ffbac237/python/pyarrow/array.pxi#L5396-L5407
and
https://github.com/apache/arrow/blob/07be48c5402adedb68c510531939a907ffbac237/python/pyarrow/pandas-shim.pxi#L231-L242
So we already have some custom handling for Period/Interval, but that will also be needed for DatetimeTZDtype
Contributor guide
Assessment
This issue has not been assessed yet.