[Python] Converting python array to TimestampArray with naive datetime and datetime with various timezones
- 主要言語
- C++
- スター
- 17.1k
- フォーク
- 4.3k
- 平均マージ
- 3日 18時間
- マージ済み PR(30日)
- 91
説明
### Describe the bug, including details regarding any error messages, version, and platform.
When converting a python array with datetime elements and mixed timezones into a pyarrow array there are two points that seem to be incorrect/could be improved:
- the values seem to be calculated to the UTC timezone but the `tz` attribute of the `timestamp` is defaulted to the timezone of the first element in an array (seems wrong to me),
- together with datetime elements with timezones a naive element can also be present and it is presumed that the naive element is in UTC timezone, which is not necessarily true.
```python
>>> import zoneinfo
>>> import datetime
>>> import pyarrow as pa
# Mixed timezones without naive datetime
>>> data_mixed = [
... datetime.datetime(2006, 1, 13, 12, 34, 56, 432539, tzinfo=zoneinfo.ZoneInfo(key='US/Eastern')),
... datetime.datetime(2008, 1, 5, 5, 0, 0, 1000, tzinfo=datetime.timezone.utc),
... datetime.datetime(2010, 8, 13, 5, 0, 0, 437699, tzinfo=zoneinfo.ZoneInfo(key='Europe/Moscow')),
... ]
>>> pa.array(data_mixed)
[
2006-01-13 17:34:56.432539,
2008-01-05 05:00:00.001000,
2010-08-13 01:00:00.437699
]
>>> pa.array(data_mixed).type
TimestampType(timestamp[us, tz=US/Eastern])
# Mixed timezones with naive datetime as the first element
>>> data_mixed_with_naive_first = [
... datetime.datetime(2007, 7, 13, 8, 23, 34, 123456), # naive
... datetime.datetime(2008, 1, 5, 5, 0, 0, 1000, tzinfo=datetime.timezone.utc),
... None,
... datetime.datetime(2006, 1, 13, 12, 34, 56, 432539, tzinfo=zoneinfo.ZoneInfo(key='US/Eastern')),
... datetime.datetime(2010, 8, 13, 5, 0, 0, 437699, tzinfo=zoneinfo.ZoneInfo(key='Europe/Moscow')),
... ]
>>> pa.array(data_mixed_with_naive_first)
[
2007-07-13 08:23:34.123456,
2008-01-05 05:00:00.001000,
null,
2006-01-13 17:34:56.432539,
2010-08-13 01:00:00.437699
]
>>> pa.array(data_mixed_with_naive_first).type
TimestampType(timestamp[us])
# Mixed timezones with naive datetime not as first element
>>> data_mixed_with_naive = [
... datetime.datetime(2006, 1, 13, 12, 34, 56, 432539, tzinfo=zoneinfo.ZoneInfo(key='US/Eastern')),
... datetime.datetime(2010, 8, 13, 5, 0, 0, 437699, tzinfo=zoneinfo.ZoneInfo(key='Europe/Moscow')),
... datetime.datetime(2008, 1, 5, 5, 0, 0, 1000, tzinfo=datetime.timezone.utc),
... datetime.datetime(2007, 7, 13, 8, 23, 34, 123456), # naive
... None,
... ]
>>> pa.array(data_mixed_with_naive)
[
2006-01-13 17:34:56.432539,
2010-08-13 01:00:00.437699,
2008-01-05 05:00:00.001000,
2007-07-13 08:23:34.123456,
null
]
>>> pa.array(data_mixed_with_naive).type
TimestampType(timestamp[us, tz=US/Eastern])
```
I think that if the datetime elements with various timezones are defaulted to UTC then we should also do the same with the `tz` attribute.
As for the case where a naive element is present the conversion could turn out an error and advise the user to add a timezone or have all elements naive.
### Component(s)
Python
コントリビューションガイド
調査の方向性
まず pa.array を使って例を再現し、Python の datetime から TimestampArray への変換経路を調査します。aware な datetime が混在する場合と、naive な datetime を含むリストの場合について、結果のタイムスタンプ値とタイムゾーンメタデータを比較します。完了条件として、混在する naive 値とタイムゾーン対応値に対して、一貫したタイムゾーンメタデータまたは明確なエラーを確立し、報告されたケースのリグレッションテストを追加します。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- python
- 領域
- data
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 停滞
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 35/100