apache / apache/arrow

[Python] Converting python array to TimestampArray with naive datetime and datetime with various timezones

オープン
#34,412 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
Component: Python Type: bug
主要言語
C++
スター
17.1k
フォーク
4.3k
平均マージ
3日 18時間
マージ済み PR(30日)
91

説明

### Describe the bug, including details regarding any error messages, version, and platform.

When converting a python array with datetime elements and mixed timezones into a pyarrow array there are two points that seem to be incorrect/could be improved:

- the values seem to be calculated to the UTC timezone but the `tz` attribute of the `timestamp` is defaulted to the timezone of the first element in an array (seems wrong to me),
- together with datetime elements with timezones a naive element can also be present and it is presumed that the naive element is in UTC timezone, which is not necessarily true.

```python
>>> import zoneinfo
>>> import datetime
>>> import pyarrow as pa

# Mixed timezones without naive datetime
>>> data_mixed = [
... datetime.datetime(2006, 1, 13, 12, 34, 56, 432539, tzinfo=zoneinfo.ZoneInfo(key='US/Eastern')),
... datetime.datetime(2008, 1, 5, 5, 0, 0, 1000, tzinfo=datetime.timezone.utc),
... datetime.datetime(2010, 8, 13, 5, 0, 0, 437699, tzinfo=zoneinfo.ZoneInfo(key='Europe/Moscow')),
... ]
>>> pa.array(data_mixed)

[
2006-01-13 17:34:56.432539,
2008-01-05 05:00:00.001000,
2010-08-13 01:00:00.437699
]
>>> pa.array(data_mixed).type
TimestampType(timestamp[us, tz=US/Eastern])

# Mixed timezones with naive datetime as the first element
>>> data_mixed_with_naive_first = [
... datetime.datetime(2007, 7, 13, 8, 23, 34, 123456), # naive
... datetime.datetime(2008, 1, 5, 5, 0, 0, 1000, tzinfo=datetime.timezone.utc),
... None,
... datetime.datetime(2006, 1, 13, 12, 34, 56, 432539, tzinfo=zoneinfo.ZoneInfo(key='US/Eastern')),
... datetime.datetime(2010, 8, 13, 5, 0, 0, 437699, tzinfo=zoneinfo.ZoneInfo(key='Europe/Moscow')),
... ]
>>> pa.array(data_mixed_with_naive_first)

[
2007-07-13 08:23:34.123456,
2008-01-05 05:00:00.001000,
null,
2006-01-13 17:34:56.432539,
2010-08-13 01:00:00.437699
]
>>> pa.array(data_mixed_with_naive_first).type
TimestampType(timestamp[us])

# Mixed timezones with naive datetime not as first element
>>> data_mixed_with_naive = [
... datetime.datetime(2006, 1, 13, 12, 34, 56, 432539, tzinfo=zoneinfo.ZoneInfo(key='US/Eastern')),
... datetime.datetime(2010, 8, 13, 5, 0, 0, 437699, tzinfo=zoneinfo.ZoneInfo(key='Europe/Moscow')),
... datetime.datetime(2008, 1, 5, 5, 0, 0, 1000, tzinfo=datetime.timezone.utc),
... datetime.datetime(2007, 7, 13, 8, 23, 34, 123456), # naive
... None,
... ]
>>> pa.array(data_mixed_with_naive)

[
2006-01-13 17:34:56.432539,
2010-08-13 01:00:00.437699,
2008-01-05 05:00:00.001000,
2007-07-13 08:23:34.123456,
null
]
>>> pa.array(data_mixed_with_naive).type
TimestampType(timestamp[us, tz=US/Eastern])
```

I think that if the datetime elements with various timezones are defaulted to UTC then we should also do the same with the `tz` attribute.

As for the case where a naive element is present the conversion could turn out an error and advise the user to add a timezone or have all elements naive.

### Component(s)

Python

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

まず pa.array を使って例を再現し、Python の datetime から TimestampArray への変換経路を調査します。aware な datetime が混在する場合と、naive な datetime を含むリストの場合について、結果のタイムスタンプ値とタイムゾーンメタデータを比較します。完了条件として、混在する naive 値とタイムゾーン対応値に対して、一貫したタイムゾーンメタデータまたは明確なエラーを確立し、報告されたケースのリグレッションテストを追加します。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python
領域
data
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
35/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。