airbytehq / airbytehq/airbyte

Reads (in both Source and Destination) should batch on bytes read instead of records read.

Open
#3,439 12 comments 11 reactions 0 assignees View on GitHub
area/reliability frozen lang/java team/db-dw-sources type/enhancement
Dominant language
Python
Stars
22.1k
Forks
5.3k
PR merge metrics
PR metrics pending

Description

## Tell us about the problem you're trying to solve
Today, we read continue reading records until we hit a batch size - currently 10k. This is fine for most cases. However, this can cause OOM errors for tables with large row size. e.g. a table with an average row size of 2MB will require a RAM of 20GB.

This is pretty simple for Destinations - since Destinations read record by record, they can check memory usage after each record and stop at a preconfigured limit to the maximum available heap size.

This is slightly trickier for Sources. Sources read data in batches - the only way to know how much memory a batch of data requires is to read the data. We'd probably need some sort of dynamic batching algorithm here, and a way to recover from memory exceptions.

## Describe the solution you’d like
This should also take into account byte size as well. e.g. Insert the records if the record or byte limit is hit, whichever comes first.

┆Issue is synchronized with this [Asana task](https://app.asana.com/0/1200367912513076/1200368197521899) by [Unito](https://www.unito.io)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.