airbytehq / airbytehq/PyAirbyte

Snowflake is slow

Đang mở
#38 1 bình luận 0 reaction 1 người được giao Được @aaronsteers nhận Xem trên GitHub
accepting pull requests
Ngôn ngữ chính
Python
Star
344
Fork
77
Merge trung bình
1 ngày 11 giờ
Pull request đã merge (30 ngày)
35

Mô tả

Relative to other cache types, Snowflake is pretty slow right now. It appears the main issue is that there are lots of calls which have a round-trip time of .8s-1.5s. These add up to runtimes close to 30-40 seconds to do a simple load.

In the context of a 10-20 minute data sync, these slowdowns may not be noticeable, but they are very noticeable in our integration tests.

For now, I've already moved all Snowflake integration tests into their own integration test file, separating them from the Postgres and DuckDB tests which are much faster.

I'm also attaching a perf trace log from viztracer, for discussion and future reference.

This test ran in ~10 seconds. You can see several calls to the database taking in the .8-1.9s range:
> ![Image](https://github.com/airbytehq/airbyte-lib-private-beta/assets/18150651/8c0943db-799a-4ee4-9496-ea402fc875e5)

[result_test_faker_read_to_snowflake_170789807552213.json.zip](https://github.com/airbytehq/airbyte-lib-private-beta/files/14284499/result_test_faker_read_to_snowflake_170789807552213.json.zip)

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.