googleapis / googleapis/google-cloud-python

Support INTERVAL data type in DB-API, arrow, and pandas connectors

未关闭
#15,739 5 条评论 1 个 reaction 已指派 1 人 已被 @tswast 认领 在 GitHub 查看
api: bigquery external status: blocked type: feature request
主要语言
Python
星标
5.4k
派生
1.8k
平均合并
3 天 4 小时
30 天内合并 PR
122

描述

Follow-up to https://github.com/googleapis/google-cloud-python/issues/15740 since pandas and Arrow do not yet have a structured type that aligns with `INTERVAL`. The existing [Timedelta](https://pandas.pydata.org/docs/reference/api/pandas.Timedelta.html) support would work for INTERVALS with only a time component, but it is not calendar aware, so supporting year, month, and day intervals would require some mapping to timedelta, which is not ideal.

Why is a new data type needed?

* YEAR: Leap years are a thing. Not every year is 365 days long.
* MONTH: Not every month is the same length.
* DAY: Daylight savings is a thing. Not every day is 24 hours long.

Note: DB-API support is included here because it uses the BigQuery Storage API, where we use the Arrow wire format.

TODO:

- [ ] Auto-detect data type in DB-API query parameters
- Might be possible to do this before reading INTERVAL columns is supported.
- [ ] Row data is converted to relevant type in DB-API
- [ ] Row data is converted to relevant type in `to_dataframe`
- Might need to be `object`, since `timedelta64` doesn't have years/months.
- [ ] Check if `to_arrow` type is expected datatype
- [ ] Convert data type in `insert_rows_from_dataframe`
- [ ] Convert data type in `load_rows_from_dataframe` (CSV)
- [ ] Convert data type in `load_rows_from_dataframe` (Parquet)

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。