googleapis / googleapis/google-cloud-python

feat(bigquery): Support queryResultsFormat and compressionCodec in query_and_wait()

Đang mở
#18,047 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
api: bigquery priority: p3 type: feature request
Ngôn ngữ chính
Python
Star
5.4k
Fork
1.8k
Merge trung bình
3 ngày 4 giờ
Pull request đã merge (30 ngày)
122

Mô tả

### Determine this is the right repository

- [x] I determined this is the correct repository in which to report this feature request.

### Summary of the feature request

### Feature Description
Add support for `query_results_format` (e.g., `"ARROW"`) and `compression_codec` (e.g., `"LZ4_FRAME"`) in `Client.query_and_wait()` and related job helper functions in `google-cloud-bigquery`.

### Use Case & Problem Statement
Currently, `Client.query_and_wait()` executes a query and fetches results, but does not allow callers to request the underlying query results format as Apache Arrow streams or specify compression options directly during the fetch process.

For high-performance data processing pipelines using Apache Arrow or PyArrow, requesting query results in Arrow format with compression significantly reduces network payload size, deserialization overhead, and total query result fetch latency.

### Proposed Solution
1. Update `google.cloud.bigquery.client.Client.query_and_wait()` to accept optional parameters:
- `query_results_format`: Optional string specifying result serialization format (e.g. `"ARROW"`).
- `compression_codec`: Optional string specifying compression algorithm
2. Update underlying helper functions in `_job_helpers.py` to populate these options in the request payload sent to the BigQuery API.
3. Update `RowIterator` and Arrow parsing logic in `table.py` to seamlessly handle Arrow record batch streams when `query_results_format="ARROW"` is specified.

### Alternative Approaches Considered
- Manually constructing custom `QueryJob` instances and extracting Arrow record batches post-execution, which requires boilerplate code and loses the convenience of `query_and_wait()`.

### Additional Context
Related PR: https://github.com/googleapis/google-cloud-python/pull/18027

### Desired code experience

from google.cloud import bigquery

client = bigquery.Client()

query = "SELECT * FROM `bigquery-public-data.usa_names.usa_1910_2013` LIMIT 10000"

# Request query results directly in Apache Arrow format with LZ4_FRAME compression
results = client.query_and_wait(
query,
query_results_format="ARROW",
compression_codec="LZ4_FRAME",
)

# Convert directly to PyArrow Table or Pandas DataFrame
arrow_table = results.to_arrow()
df = results.to_dataframe()

### Expected results

1. Client.query_and_wait() accepts query_results_format and compression_codec parameters.
2. The BigQuery API receives these parameters in the job execution request.
3. Query results are returned and parsed efficiently via compressed Apache Arrow streams into the resulting RowIterator / Arrow table without client errors.

### API client name and version

_No response_

### Use case

_e.g. This feature would be useful in a scenario where I need to process large amounts of data efficiently._

### Additional context

https://github.com/googleapis/google-cloud-python/pull/18027

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Hướng nghiên cứu

Start at Client.query_and_wait(), then trace _job_helpers.py for the BigQuery request payload and table.py for RowIterator and Arrow parsing. Compare the related PR 18027 and verify that both options are accepted, sent to the API, and that compressed Arrow results support the stated to_arrow() and to_dataframe() use cases.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
api, data
Loại issue
Tính năng
Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức độ hoạt động
Ít trao đổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
52/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.