googleapis / googleapis/google-cloud-python

Faster INSERT methods

Đang mở
#15,795 14 bình luận 4 reaction 1 người được giao Được @tswast nhận Xem trên GitHub
api: bigquery priority: p3 type: feature request
Ngôn ngữ chính
Python
Star
5.4k
Fork
1.8k
Merge trung bình
3 ngày 4 giờ
Pull request đã merge (30 ngày)
122

Mô tả

**Is your feature request related to a problem? Please describe.**
I need to work with large amounts of rows on an ETL service and wanted to use SQLAlchemy ORM backed by BigQuery dialect to write more pythonic code, but it is extremely slow to upload more than tens of rows.

**Describe the solution you'd like**
Be able to use ORM-Enabled INSERT like the SQLAlchemy docs suggest, or at least the soon-to-be legacy `Session.bulk_insert_mappings` and have better performance

**Describe alternatives you've considered**
I've tried using all methods provided by this [SQLA FAQ](https://docs.sqlalchemy.org/en/14/faq/performance.html#i-m-inserting-400-000-rows-with-the-orm-and-it-s-really-slow) and none could be faster than about 900 seconds for a minimum of 200 rows.

**Aditional context**
While the examples shows that methods like the bulk_insert_mappings could do 0.4 seconds for hundreds of thousands of rows, using BigQuery dialect I've faced 2 h 30min to upload about 4k rows. I don't expect that BQ could reach anything close to the 0.4 seconds for 100k rows, but something around 5, 6 minutes I do believe is possible, since it is the average time the BigQuery client from `google.cloud.bigquery` can do for an upload of about 90k rows.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.