DMS: Improve performance by processing in parallel
Open
- Dominant language
- Python
- Stars
- 2
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
## Thoughts
Because a standard Kinesis stream uses four shards by default, and because we have been able to work around a serialization order flaw per https://github.com/crate/commons-codec/issues/101#issuecomment-3024467188, it would be nice to consume events from those four shards (or more) in parallel, to speed up data loading.
It looks like the async-kinesis package we are using for tapping into a Kinesis stream works well for this purpose of coordinating multiple consumers.
- https://github.com/crate/cratedb-toolkit/issues/475
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.