apache / apache/arrow

[Python] Converting python array to TimestampArray with naive datetime and datetime with various timezones

Aperta
#34,412 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Component: Python Type: bug
Lingua principale
C++
Stelle
17.1k
Fork
4.3k
Merge medio
3g 20h
PR unite (30g)
94

Descrizione

### Describe the bug, including details regarding any error messages, version, and platform.

When converting a python array with datetime elements and mixed timezones into a pyarrow array there are two points that seem to be incorrect/could be improved:

- the values seem to be calculated to the UTC timezone but the `tz` attribute of the `timestamp` is defaulted to the timezone of the first element in an array (seems wrong to me),
- together with datetime elements with timezones a naive element can also be present and it is presumed that the naive element is in UTC timezone, which is not necessarily true.

```python
>>> import zoneinfo
>>> import datetime
>>> import pyarrow as pa

# Mixed timezones without naive datetime
>>> data_mixed = [
... datetime.datetime(2006, 1, 13, 12, 34, 56, 432539, tzinfo=zoneinfo.ZoneInfo(key='US/Eastern')),
... datetime.datetime(2008, 1, 5, 5, 0, 0, 1000, tzinfo=datetime.timezone.utc),
... datetime.datetime(2010, 8, 13, 5, 0, 0, 437699, tzinfo=zoneinfo.ZoneInfo(key='Europe/Moscow')),
... ]
>>> pa.array(data_mixed)

[
2006-01-13 17:34:56.432539,
2008-01-05 05:00:00.001000,
2010-08-13 01:00:00.437699
]
>>> pa.array(data_mixed).type
TimestampType(timestamp[us, tz=US/Eastern])

# Mixed timezones with naive datetime as the first element
>>> data_mixed_with_naive_first = [
... datetime.datetime(2007, 7, 13, 8, 23, 34, 123456), # naive
... datetime.datetime(2008, 1, 5, 5, 0, 0, 1000, tzinfo=datetime.timezone.utc),
... None,
... datetime.datetime(2006, 1, 13, 12, 34, 56, 432539, tzinfo=zoneinfo.ZoneInfo(key='US/Eastern')),
... datetime.datetime(2010, 8, 13, 5, 0, 0, 437699, tzinfo=zoneinfo.ZoneInfo(key='Europe/Moscow')),
... ]
>>> pa.array(data_mixed_with_naive_first)

[
2007-07-13 08:23:34.123456,
2008-01-05 05:00:00.001000,
null,
2006-01-13 17:34:56.432539,
2010-08-13 01:00:00.437699
]
>>> pa.array(data_mixed_with_naive_first).type
TimestampType(timestamp[us])

# Mixed timezones with naive datetime not as first element
>>> data_mixed_with_naive = [
... datetime.datetime(2006, 1, 13, 12, 34, 56, 432539, tzinfo=zoneinfo.ZoneInfo(key='US/Eastern')),
... datetime.datetime(2010, 8, 13, 5, 0, 0, 437699, tzinfo=zoneinfo.ZoneInfo(key='Europe/Moscow')),
... datetime.datetime(2008, 1, 5, 5, 0, 0, 1000, tzinfo=datetime.timezone.utc),
... datetime.datetime(2007, 7, 13, 8, 23, 34, 123456), # naive
... None,
... ]
>>> pa.array(data_mixed_with_naive)

[
2006-01-13 17:34:56.432539,
2010-08-13 01:00:00.437699,
2008-01-05 05:00:00.001000,
2007-07-13 08:23:34.123456,
null
]
>>> pa.array(data_mixed_with_naive).type
TimestampType(timestamp[us, tz=US/Eastern])
```

I think that if the datetime elements with various timezones are defaulted to UTC then we should also do the same with the `tz` attribute.

As for the case where a naive element is present the conversion could turn out an error and advise the user to add a timezone or have all elements naive.

### Component(s)

Python

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia riproducendo gli esempi con pa.array e ispeziona il percorso di conversione da datetime di Python a TimestampArray. Confronta i valori di timestamp risultanti e i metadati del fuso orario per datetime aware misti e per gli elenchi contenenti datetime naive. L’attività dovrebbe stabilire metadati coerenti del fuso orario o un errore chiaro per valori naive e consapevoli del fuso orario misti, con copertura di regressione per i casi segnalati.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
data
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.