crate / crate/commons-codec

DMS: Improve performance by synthesizing batches

Open
#121 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2
Forks
2
PR merge metrics
No merged PRs in 30d

Description

## Problem

CTK is currently processing DMS events sequentially, see [io/kinesis/relay.py](https://github.com/crate/cratedb-toolkit/blob/v0.0.37/cratedb_toolkit/io/kinesis/relay.py#L105-L115). The consequence is data loading will be slow.

## Proposal

Because the ingress DMS event stream can include arbitarily-alternating event types, the data loading in CTK needs a component which synthesizes per-table batches of the same kind of DML operation before submitting them to the database.

It would be nice if this package could provide such a component, because it includes and implements relevant domain knowledge about DMS events already.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.