airbytehq / airbytehq/PyAirbyte

Snowflake is slow

オープン
#38 コメント 1 件 リアクション 0 件 担当者 1 名 @aaronsteers が担当を希望しています GitHub で見る
accepting pull requests
主要言語
Python
スター
344
フォーク
77
平均マージ
1日 11時間
マージ済み PR(30日)
35

説明

Relative to other cache types, Snowflake is pretty slow right now. It appears the main issue is that there are lots of calls which have a round-trip time of .8s-1.5s. These add up to runtimes close to 30-40 seconds to do a simple load.

In the context of a 10-20 minute data sync, these slowdowns may not be noticeable, but they are very noticeable in our integration tests.

For now, I've already moved all Snowflake integration tests into their own integration test file, separating them from the Postgres and DuckDB tests which are much faster.

I'm also attaching a perf trace log from viztracer, for discussion and future reference.

This test ran in ~10 seconds. You can see several calls to the database taking in the .8-1.9s range:
> ![Image](https://github.com/airbytehq/airbyte-lib-private-beta/assets/18150651/8c0943db-799a-4ee4-9496-ea402fc875e5)

[result_test_faker_read_to_snowflake_170789807552213.json.zip](https://github.com/airbytehq/airbyte-lib-private-beta/files/14284499/result_test_faker_read_to_snowflake_170789807552213.json.zip)

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。