Source CockroachDB: fails at >10M records
- Lingua principale
- Python
- Stelle
- 22.1k
- Fork
- 5.3k
- Metriche di merge delle PR
- Metriche PR in attesa
Descrizione
## Environment
- **Airbyte version**: 0.39.7-alpha and 0.39.5-alpha
- **OS Version / Instance**: Ubuntu 18.04.5 LTS
- **Deployment**: Docker-Compose (directly from airbyte repo)
- **Source Connector and version**: (if applicable example Salesforce 0.2.3) CockroachDB 0.1.12
- **Destination Connector and version**: (if applicable example Postgres 0.3.3) Postgres 0.3.20
- **Step where error happened**: Sync job
## Current Behavior
When importing a 18.1Gb (57M Records) table from CRDB using the Cockroach connector, the sync job fails. Before failing, the adapter fills all available RAM before the container shuts down: (presumably an OOM).
This does not happen on smaller tables with about 1M records, with the same source and destination:

## Expected Behavior
Airbyte should
1. not throw OOM when transferring the data
2. allow for manual setting of batch size if dynamic batch size fails
## Logs

[logs-5.txt](https://github.com/airbytehq/airbyte/files/8824188/logs-5.txt)
## Steps to Reproduce
1. Spin up airbyte, postgres and crdb (open source containers) as described in respective docs
1. Populate CRDB
1. Connect CRDB cluster to airbyte
1. attempt to transfer a large table (20gB or 60M records)
## Are you willing to submit a PR?
I have no experience with Java, but any Python / Go code we can work with.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.