airbytehq / airbytehq/airbyte

Source CockroachDB: fails at >10M records

Aberta
#13,421 5 comentários 1 reação 0 responsáveis Ver no GitHub
community connectors/source/cockroachdb frozen team/db-dw-sources type/bug
Linguagem predominante
Python
Estrelas
22.1k
Forks
5.4k
Merge médio
5h
PRs com merge (30d)
671

Descrição

## Environment
- **Airbyte version**: 0.39.7-alpha and 0.39.5-alpha
- **OS Version / Instance**: Ubuntu 18.04.5 LTS
- **Deployment**: Docker-Compose (directly from airbyte repo)
- **Source Connector and version**: (if applicable example Salesforce 0.2.3) CockroachDB 0.1.12
- **Destination Connector and version**: (if applicable example Postgres 0.3.3) Postgres 0.3.20
- **Step where error happened**: Sync job

## Current Behavior
When importing a 18.1Gb (57M Records) table from CRDB using the Cockroach connector, the sync job fails. Before failing, the adapter fills all available RAM before the container shuts down: (presumably an OOM).

This does not happen on smaller tables with about 1M records, with the same source and destination:
Screenshot 2022-06-02 at 15 24 02

## Expected Behavior
Airbyte should
1. not throw OOM when transferring the data
2. allow for manual setting of batch size if dynamic batch size fails

## Logs

![image](https://user-images.githubusercontent.com/29195438/171638991-c5d379c6-2553-4540-8c1f-530726f6726f.png)

[logs-5.txt](https://github.com/airbytehq/airbyte/files/8824188/logs-5.txt)

## Steps to Reproduce
1. Spin up airbyte, postgres and crdb (open source containers) as described in respective docs
1. Populate CRDB
1. Connect CRDB cluster to airbyte
1. attempt to transfer a large table (20gB or 60M records)

## Are you willing to submit a PR?
I have no experience with Java, but any Python / Go code we can work with.

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.