apache / apache/arrow

[Python] Incorrect conversion from datetime.datetime and datetime.time objects with non-UTC timezones

Open
#40,122 3 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I may be missing something fundamental, but Pyarrow seems to improperly handle the conversion from datetime.datetime and datetime.time objects with non-UTC timezones.

**Datetimes**

This Python datetime is the turn of the millenium in UTC+1:

```python
>>> dt = datetime.datetime(2000, 1, 1, 0, 0, tzinfo=datetime.timezone(datetime.timedelta(hours=1)))
```

Converting to pyarrow makes it 11 PM in UTC+1, which is a different time (not just the same time in a different coordinate system):

```python
>>> a = pa.array([dt])
>>> a

[
1999-12-31 23:00:00.000000
]
>>> a.type
TimestampType(timestamp[us, tz=+01:00])
```

When converted to R (using the Arrow C API), this becomes the following POSIXct vector:
```R
> a
[1] "1999-12-31 23:00:00"
> attr(a, 'tzone')
[1] "+01:00"
```

which is again 11 PM in UTC+1.

So the Arrow -> R conversion is consistent, but the Python -> Arrow conversion seems like a bug.

**Times**

Creating a pyarrow array from a datetime.time strips timezone info:

```python
>>> pa.array([datetime.time(12, 0, tzinfo=datetime.timezone(datetime.timedelta(hours=1)))]) == \
... pa.array([datetime.time(12, 0)])
True
```

which seems to be intentional according to https://arrow.apache.org/docs/python/generated/pyarrow.Time64Array.html:

> Localized timestamps will currently be returned as UTC (pandas’s native representation). Timezone-naive data will be implicitly interpreted as UTC.

But again, these are two different times (not just the same time in a different coordinate system), so they shouldn't compare equal.

This is on pyarrow 14.0.2, Python 3.12, R version 4.3.2, R Arrow version 14.0.1.

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.