googleapis / googleapis/google-cloud-python

Support INTERVAL data type in DB-API, arrow, and pandas connectors

Aperta
#15,739 5 commenti 1 reazione 1 assegnatario Rivendicata da @tswast Vedi su GitHub
api: bigquery external status: blocked type: feature request
Lingua principale
Python
Stelle
5.4k
Fork
1.8k
Merge medio
3g 4h
PR unite (30g)
122

Descrizione

Follow-up to https://github.com/googleapis/google-cloud-python/issues/15740 since pandas and Arrow do not yet have a structured type that aligns with `INTERVAL`. The existing [Timedelta](https://pandas.pydata.org/docs/reference/api/pandas.Timedelta.html) support would work for INTERVALS with only a time component, but it is not calendar aware, so supporting year, month, and day intervals would require some mapping to timedelta, which is not ideal.

Why is a new data type needed?

* YEAR: Leap years are a thing. Not every year is 365 days long.
* MONTH: Not every month is the same length.
* DAY: Daylight savings is a thing. Not every day is 24 hours long.

Note: DB-API support is included here because it uses the BigQuery Storage API, where we use the Arrow wire format.

TODO:

- [ ] Auto-detect data type in DB-API query parameters
- Might be possible to do this before reading INTERVAL columns is supported.
- [ ] Row data is converted to relevant type in DB-API
- [ ] Row data is converted to relevant type in `to_dataframe`
- Might need to be `object`, since `timedelta64` doesn't have years/months.
- [ ] Check if `to_arrow` type is expected datatype
- [ ] Convert data type in `insert_rows_from_dataframe`
- [ ] Convert data type in `load_rows_from_dataframe` (CSV)
- [ ] Convert data type in `load_rows_from_dataframe` (Parquet)

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.