Source CockroachDB: fails at >10M records
- Linguagem predominante
- Python
- Estrelas
- 22.1k
- Forks
- 5.4k
- Merge médio
- 5h
- PRs com merge (30d)
- 671
Descrição
## Environment
- **Airbyte version**: 0.39.7-alpha and 0.39.5-alpha
- **OS Version / Instance**: Ubuntu 18.04.5 LTS
- **Deployment**: Docker-Compose (directly from airbyte repo)
- **Source Connector and version**: (if applicable example Salesforce 0.2.3) CockroachDB 0.1.12
- **Destination Connector and version**: (if applicable example Postgres 0.3.3) Postgres 0.3.20
- **Step where error happened**: Sync job
## Current Behavior
When importing a 18.1Gb (57M Records) table from CRDB using the Cockroach connector, the sync job fails. Before failing, the adapter fills all available RAM before the container shuts down: (presumably an OOM).
This does not happen on smaller tables with about 1M records, with the same source and destination:

## Expected Behavior
Airbyte should
1. not throw OOM when transferring the data
2. allow for manual setting of batch size if dynamic batch size fails
## Logs

[logs-5.txt](https://github.com/airbytehq/airbyte/files/8824188/logs-5.txt)
## Steps to Reproduce
1. Spin up airbyte, postgres and crdb (open source containers) as described in respective docs
1. Populate CRDB
1. Connect CRDB cluster to airbyte
1. attempt to transfer a large table (20gB or 60M records)
## Are you willing to submit a PR?
I have no experience with Java, but any Python / Go code we can work with.
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.