[Python] Incorrect conversion from datetime.datetime and datetime.time objects with non-UTC timezones
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I may be missing something fundamental, but Pyarrow seems to improperly handle the conversion from datetime.datetime and datetime.time objects with non-UTC timezones.
**Datetimes**
This Python datetime is the turn of the millenium in UTC+1:
```python
>>> dt = datetime.datetime(2000, 1, 1, 0, 0, tzinfo=datetime.timezone(datetime.timedelta(hours=1)))
```
Converting to pyarrow makes it 11 PM in UTC+1, which is a different time (not just the same time in a different coordinate system):
```python
>>> a = pa.array([dt])
>>> a
[
1999-12-31 23:00:00.000000
]
>>> a.type
TimestampType(timestamp[us, tz=+01:00])
```
When converted to R (using the Arrow C API), this becomes the following POSIXct vector:
```R
> a
[1] "1999-12-31 23:00:00"
> attr(a, 'tzone')
[1] "+01:00"
```
which is again 11 PM in UTC+1.
So the Arrow -> R conversion is consistent, but the Python -> Arrow conversion seems like a bug.
**Times**
Creating a pyarrow array from a datetime.time strips timezone info:
```python
>>> pa.array([datetime.time(12, 0, tzinfo=datetime.timezone(datetime.timedelta(hours=1)))]) == \
... pa.array([datetime.time(12, 0)])
True
```
which seems to be intentional according to https://arrow.apache.org/docs/python/generated/pyarrow.Time64Array.html:
> Localized timestamps will currently be returned as UTC (pandas’s native representation). Timezone-naive data will be implicitly interpreted as UTC.
But again, these are two different times (not just the same time in a different coordinate system), so they shouldn't compare equal.
This is on pyarrow 14.0.2, Python 3.12, R version 4.3.2, R Arrow version 14.0.1.
### Component(s)
Python
Contributor guide
Assessment
This issue has not been assessed yet.