googleapis / googleapis/google-cloud-python
Support INTERVAL data type in DB-API, arrow, and pandas connectors
- 主要语言
- Python
- 星标
- 5.4k
- 派生
- 1.8k
- 平均合并
- 3 天 4 小时
- 30 天内合并 PR
- 122
描述
Follow-up to https://github.com/googleapis/google-cloud-python/issues/15740 since pandas and Arrow do not yet have a structured type that aligns with `INTERVAL`. The existing [Timedelta](https://pandas.pydata.org/docs/reference/api/pandas.Timedelta.html) support would work for INTERVALS with only a time component, but it is not calendar aware, so supporting year, month, and day intervals would require some mapping to timedelta, which is not ideal.
Why is a new data type needed?
* YEAR: Leap years are a thing. Not every year is 365 days long.
* MONTH: Not every month is the same length.
* DAY: Daylight savings is a thing. Not every day is 24 hours long.
Note: DB-API support is included here because it uses the BigQuery Storage API, where we use the Arrow wire format.
TODO:
- [ ] Auto-detect data type in DB-API query parameters
- Might be possible to do this before reading INTERVAL columns is supported.
- [ ] Row data is converted to relevant type in DB-API
- [ ] Row data is converted to relevant type in `to_dataframe`
- Might need to be `object`, since `timedelta64` doesn't have years/months.
- [ ] Check if `to_arrow` type is expected datatype
- [ ] Convert data type in `insert_rows_from_dataframe`
- [ ] Convert data type in `load_rows_from_dataframe` (CSV)
- [ ] Convert data type in `load_rows_from_dataframe` (Parquet)
贡献指南
评估
这个 Issue 还没有评估数据。