googleapis / googleapis/google-cloud-python

Support INTERVAL data type in DB-API, arrow, and pandas connectors

Abierto
#15,739 5 comentarios 1 reacción 1 asignado Reclamado por @tswast Ver en GitHub
api: bigquery external status: blocked type: feature request
Lenguaje dominante
Python
Estrellas
5.4k
Forks
1.8k
Merge medio
3 d 4 h
PR fusionados (30 d)
122

Descripción

Follow-up to https://github.com/googleapis/google-cloud-python/issues/15740 since pandas and Arrow do not yet have a structured type that aligns with `INTERVAL`. The existing [Timedelta](https://pandas.pydata.org/docs/reference/api/pandas.Timedelta.html) support would work for INTERVALS with only a time component, but it is not calendar aware, so supporting year, month, and day intervals would require some mapping to timedelta, which is not ideal.

Why is a new data type needed?

* YEAR: Leap years are a thing. Not every year is 365 days long.
* MONTH: Not every month is the same length.
* DAY: Daylight savings is a thing. Not every day is 24 hours long.

Note: DB-API support is included here because it uses the BigQuery Storage API, where we use the Arrow wire format.

TODO:

- [ ] Auto-detect data type in DB-API query parameters
- Might be possible to do this before reading INTERVAL columns is supported.
- [ ] Row data is converted to relevant type in DB-API
- [ ] Row data is converted to relevant type in `to_dataframe`
- Might need to be `object`, since `timedelta64` doesn't have years/months.
- [ ] Check if `to_arrow` type is expected datatype
- [ ] Convert data type in `insert_rows_from_dataframe`
- [ ] Convert data type in `load_rows_from_dataframe` (CSV)
- [ ] Convert data type in `load_rows_from_dataframe` (Parquet)

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.