crate / crate/commons-codec

DMS: Improve performance by processing in parallel

Open
#122 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2
Forks
2
PR merge metrics
No merged PRs in 30d

Description

## Thoughts

Because a standard Kinesis stream uses four shards by default, and because we have been able to work around a serialization order flaw per https://github.com/crate/commons-codec/issues/101#issuecomment-3024467188, it would be nice to consume events from those four shards (or more) in parallel, to speed up data loading.

It looks like the async-kinesis package we are using for tapping into a Kinesis stream works well for this purpose of coordinating multiple consumers.

- https://github.com/crate/cratedb-toolkit/issues/475

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.