Performance: pipeline rate always lower than the source rate
- Dominant language
- Go
- Stars
- 610
- Forks
- 63
- Avg merge
- 12h 28m
- Merged PRs (30d)
- 57
Description
### Bug description
It appears that the pipeline rate is always lower than the generator source rate. For example, when source generates records at 15k msg/s, the pipeline rate (i.e. the number of records flowing through the pipeline, acknowledged ones) is around 10k, which gives the impression that 10k msg/s is what Conduit can handle.
However, in the 10k msg/s test, the pipeline rate is around 6700 msg/s.
### Steps to reproduce
1. Run https://github.com/ConduitIO/streaming-benchmarks/blob/haris/ec2-helpers/workloads/small-messages-15k-msg-per-sec.sh
2. Run https://github.com/ConduitIO/streaming-benchmarks/blob/haris/ec2-helpers/workloads/small-messages-10k-msg-per-sec.sh
3. Compare results
### Version
{ "version": "v0.3.0-nightly.20220811", "os": "linux", "arch": "amd64" }
Contributor guide
Assessment
This issue has not been assessed yet.