googleapis / googleapis/google-cloud-python

support the Arrow PyCapsule Interface without a pyarrow dependency

Open
#15,699 3 comments 3 reactions 1 assignee Claimed by @tswast View on GitHub
api: bigquery
Dominant language
Python
Stars
5.4k
Forks
1.8k
Avg merge
3d 4h
Merged PRs (30d)
122

Description

**Is your feature request related to a problem? Please describe.**

The polars community uses the arrow in-memory format without a need for a dependency on the heavy weight pyarrow package, but the easiest way to get arrow out of and into BigQuery with this client requires the use of the pyarrow dependency.

**Describe the solution you'd like**

Expose the Arrow PyCapsule Interface on RowIterator (returned by QueryJob.result(), Client.query_and_wait(), Client.list_rows()). This functionality should not require pyarrow. See: https://gist.github.com/tswast/99b017b20386e324f5c7d2bd49f21b5f for a related example of how this could be done.

**Describe alternatives you've considered**

* Separate package exposing the Arrow PyCapsule Interface
* Always require pyarrow
* Directly integrate into polars (in-progress)

**Additional context**

* [[Python] Promote usage of the Arrow PyCapsule Protocol (for the C Data Inteface) #39195](https://github.com/apache/arrow/issues/39195)
* [support adbc APIs in DB-API package](https://github.com/googleapis/python-bigquery/issues/2015#top)
* [Support inserting data into BigQuery directly from a Polars DataFrame](https://github.com/googleapis/google-cloud-python/issues/15710#top)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.