Reads (in both Source and Destination) should batch on bytes read instead of records read.
- Dominant language
- Python
- Stars
- 22.1k
- Forks
- 5.3k
- PR merge metrics
- PR metrics pending
Description
## Tell us about the problem you're trying to solve
Today, we read continue reading records until we hit a batch size - currently 10k. This is fine for most cases. However, this can cause OOM errors for tables with large row size. e.g. a table with an average row size of 2MB will require a RAM of 20GB.
This is pretty simple for Destinations - since Destinations read record by record, they can check memory usage after each record and stop at a preconfigured limit to the maximum available heap size.
This is slightly trickier for Sources. Sources read data in batches - the only way to know how much memory a batch of data requires is to read the data. We'd probably need some sort of dynamic batching algorithm here, and a way to recover from memory exceptions.
## Describe the solution you’d like
This should also take into account byte size as well. e.g. Insert the records if the record or byte limit is hit, whichever comes first.
┆Issue is synchronized with this [Asana task](https://app.asana.com/0/1200367912513076/1200368197521899) by [Unito](https://www.unito.io)
Contributor guide
Assessment
This issue has not been assessed yet.