googleapis / googleapis/google-cloud-python

feat!: convert arrow data types to Python types in `rows()`

Offen
#14,279 3 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @tswast Auf GitHub ansehen
api: bigquerystorage next major: breaking change type: feature request
Vorherrschende Sprache
Python
Sterne
5.4k
Forks
1.8k
Ø Merge
3 T. 4 Std.
Gemergte PRs (30 T.)
122

Beschreibung

I notice we just loop through all the rows in a record batch here:

https://github.com/googleapis/python-bigquery-storage/blob/f7fc4ec45e62739139bf865497ce2e5fbd948d31/google/cloud/bigquery_storage_v1/reader.py#L695

This might result in some odd types, such as [Int64Scalar](https://arrow.apache.org/docs/python/generated/pyarrow.Int64Scalar.html?highlight=scalar#pyarrow.Int64Scalar) and [TimestampScalar](https://arrow.apache.org/docs/python/generated/pyarrow.TimestampScalar.html?highlight=scalar#pyarrow.TimestampScalar) sneaking in.

I think we probably want to call [to_pydict](https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatch.html?highlight=recordbatch#pyarrow.RecordBatch.to_pydict) before looping over rows/columns in the BigQuery Storage API client.

I'm filing this issue in `python-bigquery` at first, because I think it's worth investigating if this is actually happening first.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.