googleapis / googleapis/google-cloud-python

BigQuery: make jobs awaitable

Đang mở
#15,742 58 bình luận 41 reaction 1 người được giao Được @tswast nhận Xem trên GitHub
api: bigquery Python 3 Only type: feature request
Ngôn ngữ chính
Python
Star
5.4k
Fork
1.8k
Merge trung bình
3 ngày 4 giờ
Pull request đã merge (30 ngày)
122

Mô tả

I know BigQuery jobs are asynchronous by default. However, I am struggling to make my datapipeline async end-to-end.

Looking at [this JS example](https://medium.com/skyshidigital/what-bigquery-is-and-isnt-d092fa173d53), I thought it would be the most Pythonic to make a BigQuery job awaitable. However, I can't get that to work in Python i.e. errors when `await client.query(query)`. Looking at the source code, I don't see which method [returns an awaitable object](https://www.python.org/dev/peps/pep-0492/).

I have little experience in writing async Python code and found [this example that wraps jobs in a `async def coroutine`](https://stackoverflow.com/questions/53065340/python-how-to-big-query-asynchronous-tasks).
```
class BQApi(object):
def __init__(self):
self.api = bigquery.Client.from_service_account_json(BQ_CONFIG["credentials"])

async def exec_query(self, query, **kwargs) -> bigquery.table.RowIterator:
job = self.api.query(query, **kwargs)
task = asyncio.create_task(self.coroutine_job(job))
return await task

@staticmethod
async def coroutine_job(job):
return job.result()
```

The [`google.api_core.operation.Operation`](https://googleapis.github.io/google-cloud-python/latest/core/operation.html) shows how to use `add_done_callback` to asynchronously wait for long-running operations. I have tried that, but the following yields `AttributeError: 'QueryJob' object has no attribute '_condition'` :

```
from concurrent.futures import ThreadPoolExecutor, as_completed
query1 = 'SELECT 1'
query2 = 'SELECT 2'

def my_callback(future):
result = future.result()

operations = [bq.query(query1), bq.query(query2)]
[operation.add_done_callback(my_callback) for operation in operations]
results2 = []
for future in as_completed(operations):
results2.append(list(future.result()))
```

Given that jobs are already asynchronous, would it make sense to add a method that returns an awaitable?

Or am I missing something and is there an Pythonic way to use the BigQuery client with the async/await pattern?

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.