ConduitIO / ConduitIO/conduit

Performance: pipeline rate always lower than the source rate

Open
#571 6 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Go
Stars
610
Forks
63
Avg merge
12h 28m
Merged PRs (30d)
57

Description

### Bug description

It appears that the pipeline rate is always lower than the generator source rate. For example, when source generates records at 15k msg/s, the pipeline rate (i.e. the number of records flowing through the pipeline, acknowledged ones) is around 10k, which gives the impression that 10k msg/s is what Conduit can handle.

However, in the 10k msg/s test, the pipeline rate is around 6700 msg/s.

### Steps to reproduce

1. Run https://github.com/ConduitIO/streaming-benchmarks/blob/haris/ec2-helpers/workloads/small-messages-15k-msg-per-sec.sh
2. Run https://github.com/ConduitIO/streaming-benchmarks/blob/haris/ec2-helpers/workloads/small-messages-10k-msg-per-sec.sh
3. Compare results

### Version

{ "version": "v0.3.0-nightly.20220811", "os": "linux", "arch": "amd64" }

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.