DMS: Improve performance by synthesizing batches
- Dominant language
- Python
- Stars
- 2
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
CTK is currently processing DMS events sequentially, see [io/kinesis/relay.py](https://github.com/crate/cratedb-toolkit/blob/v0.0.37/cratedb_toolkit/io/kinesis/relay.py#L105-L115). The consequence is data loading will be slow.
## Proposal
Because the ingress DMS event stream can include arbitarily-alternating event types, the data loading in CTK needs a component which synthesizes per-table batches of the same kind of DML operation before submitting them to the database.
It would be nice if this package could provide such a component, because it includes and implements relevant domain knowledge about DMS events already.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.