airbytehq / airbytehq/PyAirbyte

Snowflake is slow

未关闭
#38 1 条评论 0 个 reaction 已指派 1 人 已被 @aaronsteers 认领 在 GitHub 查看
accepting pull requests
主要语言
Python
星标
344
派生
77
平均合并
1 天 11 小时
30 天内合并 PR
35

描述

Relative to other cache types, Snowflake is pretty slow right now. It appears the main issue is that there are lots of calls which have a round-trip time of .8s-1.5s. These add up to runtimes close to 30-40 seconds to do a simple load.

In the context of a 10-20 minute data sync, these slowdowns may not be noticeable, but they are very noticeable in our integration tests.

For now, I've already moved all Snowflake integration tests into their own integration test file, separating them from the Postgres and DuckDB tests which are much faster.

I'm also attaching a perf trace log from viztracer, for discussion and future reference.

This test ran in ~10 seconds. You can see several calls to the database taking in the .8-1.9s range:
> ![Image](https://github.com/airbytehq/airbyte-lib-private-beta/assets/18150651/8c0943db-799a-4ee4-9496-ea402fc875e5)

[result_test_faker_read_to_snowflake_170789807552213.json.zip](https://github.com/airbytehq/airbyte-lib-private-beta/files/14284499/result_test_faker_read_to_snowflake_170789807552213.json.zip)

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。