googleapis / googleapis/google-cloud-python

feat!: convert arrow data types to Python types in `rows()`

Open
#14,279 3 comments 0 reactions 1 assignee Claimed by @tswast View on GitHub
api: bigquerystorage next major: breaking change type: feature request
Dominant language
Python
Stars
5.4k
Forks
1.8k
Avg merge
3d 4h
Merged PRs (30d)
122

Description

I notice we just loop through all the rows in a record batch here:

https://github.com/googleapis/python-bigquery-storage/blob/f7fc4ec45e62739139bf865497ce2e5fbd948d31/google/cloud/bigquery_storage_v1/reader.py#L695

This might result in some odd types, such as [Int64Scalar](https://arrow.apache.org/docs/python/generated/pyarrow.Int64Scalar.html?highlight=scalar#pyarrow.Int64Scalar) and [TimestampScalar](https://arrow.apache.org/docs/python/generated/pyarrow.TimestampScalar.html?highlight=scalar#pyarrow.TimestampScalar) sneaking in.

I think we probably want to call [to_pydict](https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatch.html?highlight=recordbatch#pyarrow.RecordBatch.to_pydict) before looping over rows/columns in the BigQuery Storage API client.

I'm filing this issue in `python-bigquery` at first, because I think it's worth investigating if this is actually happening first.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.