airbytehq / airbytehq/airbyte

Databricks destination is able to transfer less than 2MB/s

Offen
#21,318 3 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
autoteam community frozen team/destinations type/bug
Vorherrschende Sprache
Python
Sterne
22.1k
Forks
5.3k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

## Environment
- OS Version / Instance: Linux (ubuntu 20.04) (Azure VM)
- Memory / Disk: 8Gb / 32 Gb
- Deployment: Docker
- Airbyte Version: 0.40.26
- Source name/version: S3 version 0.1.27
- Destination name/version: Databricks version 0.3.1
- Step: sync

## Current Behavior
I am currently trying to set up a connection between AWS S3 and Databricks on Azure but the sync is very slow.

The sync takes approx. 199s for ca. 297 MB and 200 000 rows which means a transfer rate of ~1,49MB/s and ~1500rows/s. I also tried a larger file with 2GB which took approx. 1h 15min which is a similar rate. Is this in the order of the expected transfer speed? It seems very slow to me. I also tried to speed it up by increasing the block size but I didn’t notice any change in speed when trying different block sizes. The reported speeds used a block size of 1 000 000. This would mean when I am trying to transfer 50GB per day and for some reason there is a bug and the transfer is not running for a day, I would always be a day behind because I would never be able to get up to speed. Is airbyte not designed for such big dataloads?

To be sure that it is not a bottleneck of source or destination, I also tried Faker as source which is even slower (~0.97MB/s) and S3 as destination which is in the same order (~1,61MB/s). I also monitored CPU, memory and network usage. The 4 Core VM I tried was at 50% capacity at its peak, memory was always more available and the network had the same rate as the general transfer speed which should not be the bottleneck when running on Azure.

## Expected Behavior
I would expect the transfer speed to be determined by some bottleneck like the CPU capacity or the network but it does not seem to be any of those.

## Suggestion
Maybe it would be a good idea to think about some kind of batching mechansim like in [meltano](https://meltano.com/blog/6x-more-speed-for-your-data-pipelines-with-batch-messages/)

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.