airbytehq / airbytehq/PyAirbyte

Snowflake is slow

Open
#38 1 comment 0 reactions 1 assignee Claimed by @aaronsteers View on GitHub
accepting pull requests
Dominant language
Python
Stars
344
Forks
77
Avg merge
1d 11h
Merged PRs (30d)
35

Description

Relative to other cache types, Snowflake is pretty slow right now. It appears the main issue is that there are lots of calls which have a round-trip time of .8s-1.5s. These add up to runtimes close to 30-40 seconds to do a simple load.

In the context of a 10-20 minute data sync, these slowdowns may not be noticeable, but they are very noticeable in our integration tests.

For now, I've already moved all Snowflake integration tests into their own integration test file, separating them from the Postgres and DuckDB tests which are much faster.

I'm also attaching a perf trace log from viztracer, for discussion and future reference.

This test ran in ~10 seconds. You can see several calls to the database taking in the .8-1.9s range:
> ![Image](https://github.com/airbytehq/airbyte-lib-private-beta/assets/18150651/8c0943db-799a-4ee4-9496-ea402fc875e5)

[result_test_faker_read_to_snowflake_170789807552213.json.zip](https://github.com/airbytehq/airbyte-lib-private-beta/files/14284499/result_test_faker_read_to_snowflake_170789807552213.json.zip)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.