apache / apache/arrow

[Python] Converting python array to TimestampArray with naive datetime and datetime with various timezones

未关闭
#34,412 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Component: Python Type: bug
主要语言
C++
星标
17.1k
派生
4.3k
平均合并
3 天 18 小时
30 天内合并 PR
91

描述

### Describe the bug, including details regarding any error messages, version, and platform.

When converting a python array with datetime elements and mixed timezones into a pyarrow array there are two points that seem to be incorrect/could be improved:

- the values seem to be calculated to the UTC timezone but the `tz` attribute of the `timestamp` is defaulted to the timezone of the first element in an array (seems wrong to me),
- together with datetime elements with timezones a naive element can also be present and it is presumed that the naive element is in UTC timezone, which is not necessarily true.

```python
>>> import zoneinfo
>>> import datetime
>>> import pyarrow as pa

# Mixed timezones without naive datetime
>>> data_mixed = [
... datetime.datetime(2006, 1, 13, 12, 34, 56, 432539, tzinfo=zoneinfo.ZoneInfo(key='US/Eastern')),
... datetime.datetime(2008, 1, 5, 5, 0, 0, 1000, tzinfo=datetime.timezone.utc),
... datetime.datetime(2010, 8, 13, 5, 0, 0, 437699, tzinfo=zoneinfo.ZoneInfo(key='Europe/Moscow')),
... ]
>>> pa.array(data_mixed)

[
2006-01-13 17:34:56.432539,
2008-01-05 05:00:00.001000,
2010-08-13 01:00:00.437699
]
>>> pa.array(data_mixed).type
TimestampType(timestamp[us, tz=US/Eastern])

# Mixed timezones with naive datetime as the first element
>>> data_mixed_with_naive_first = [
... datetime.datetime(2007, 7, 13, 8, 23, 34, 123456), # naive
... datetime.datetime(2008, 1, 5, 5, 0, 0, 1000, tzinfo=datetime.timezone.utc),
... None,
... datetime.datetime(2006, 1, 13, 12, 34, 56, 432539, tzinfo=zoneinfo.ZoneInfo(key='US/Eastern')),
... datetime.datetime(2010, 8, 13, 5, 0, 0, 437699, tzinfo=zoneinfo.ZoneInfo(key='Europe/Moscow')),
... ]
>>> pa.array(data_mixed_with_naive_first)

[
2007-07-13 08:23:34.123456,
2008-01-05 05:00:00.001000,
null,
2006-01-13 17:34:56.432539,
2010-08-13 01:00:00.437699
]
>>> pa.array(data_mixed_with_naive_first).type
TimestampType(timestamp[us])

# Mixed timezones with naive datetime not as first element
>>> data_mixed_with_naive = [
... datetime.datetime(2006, 1, 13, 12, 34, 56, 432539, tzinfo=zoneinfo.ZoneInfo(key='US/Eastern')),
... datetime.datetime(2010, 8, 13, 5, 0, 0, 437699, tzinfo=zoneinfo.ZoneInfo(key='Europe/Moscow')),
... datetime.datetime(2008, 1, 5, 5, 0, 0, 1000, tzinfo=datetime.timezone.utc),
... datetime.datetime(2007, 7, 13, 8, 23, 34, 123456), # naive
... None,
... ]
>>> pa.array(data_mixed_with_naive)

[
2006-01-13 17:34:56.432539,
2010-08-13 01:00:00.437699,
2008-01-05 05:00:00.001000,
2007-07-13 08:23:34.123456,
null
]
>>> pa.array(data_mixed_with_naive).type
TimestampType(timestamp[us, tz=US/Eastern])
```

I think that if the datetime elements with various timezones are defaulted to UTC then we should also do the same with the `tz` attribute.

As for the case where a naive element is present the conversion could turn out an error and advise the user to add a timezone or have all elements naive.

### Component(s)

Python

贡献指南

打开贡献指南

调研方向

首先使用 pa.array 重现这些示例,并检查从 Python datetime 到 TimestampArray 的转换路径。比较混合 aware datetime 以及包含 naive datetime 的列表所产生的时间戳值和时区元数据。完成后应为混合的 naive 值和带时区值建立一致的时区元数据,或给出明确的错误,并为报告的案例添加回归覆盖。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
data
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。