googleapis / googleapis/google-cloud-python
feat!: convert arrow data types to Python types in `rows()`
- Lingua principale
- Python
- Stelle
- 5.4k
- Fork
- 1.8k
- Merge medio
- 3g 4h
- PR unite (30g)
- 122
Descrizione
I notice we just loop through all the rows in a record batch here:
https://github.com/googleapis/python-bigquery-storage/blob/f7fc4ec45e62739139bf865497ce2e5fbd948d31/google/cloud/bigquery_storage_v1/reader.py#L695
This might result in some odd types, such as [Int64Scalar](https://arrow.apache.org/docs/python/generated/pyarrow.Int64Scalar.html?highlight=scalar#pyarrow.Int64Scalar) and [TimestampScalar](https://arrow.apache.org/docs/python/generated/pyarrow.TimestampScalar.html?highlight=scalar#pyarrow.TimestampScalar) sneaking in.
I think we probably want to call [to_pydict](https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatch.html?highlight=recordbatch#pyarrow.RecordBatch.to_pydict) before looping over rows/columns in the BigQuery Storage API client.
I'm filing this issue in `python-bigquery` at first, because I think it's worth investigating if this is actually happening first.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.