airbytehq / airbytehq/airbyte

Source CockroachDB: fails at >10M records

Aperta
#13,421 5 commenti 1 reazione 0 assegnatari Vedi su GitHub
community connectors/source/cockroachdb frozen team/db-dw-sources type/bug
Lingua principale
Python
Stelle
22.1k
Fork
5.3k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

## Environment
- **Airbyte version**: 0.39.7-alpha and 0.39.5-alpha
- **OS Version / Instance**: Ubuntu 18.04.5 LTS
- **Deployment**: Docker-Compose (directly from airbyte repo)
- **Source Connector and version**: (if applicable example Salesforce 0.2.3) CockroachDB 0.1.12
- **Destination Connector and version**: (if applicable example Postgres 0.3.3) Postgres 0.3.20
- **Step where error happened**: Sync job

## Current Behavior
When importing a 18.1Gb (57M Records) table from CRDB using the Cockroach connector, the sync job fails. Before failing, the adapter fills all available RAM before the container shuts down: (presumably an OOM).

This does not happen on smaller tables with about 1M records, with the same source and destination:
Screenshot 2022-06-02 at 15 24 02

## Expected Behavior
Airbyte should
1. not throw OOM when transferring the data
2. allow for manual setting of batch size if dynamic batch size fails

## Logs

![image](https://user-images.githubusercontent.com/29195438/171638991-c5d379c6-2553-4540-8c1f-530726f6726f.png)

[logs-5.txt](https://github.com/airbytehq/airbyte/files/8824188/logs-5.txt)

## Steps to Reproduce
1. Spin up airbyte, postgres and crdb (open source containers) as described in respective docs
1. Populate CRDB
1. Connect CRDB cluster to airbyte
1. attempt to transfer a large table (20gB or 60M records)

## Are you willing to submit a PR?
I have no experience with Java, but any Python / Go code we can work with.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.