Beam slowness compared to flink-native
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
I tried to compare a very simple beam pipeline with an equivalent flink-native pipeline. Both pipelines read strings from one kafka topic and write them to another topic. I ran each pipeline separately on a single task manager with a single slot and parallelism 1.
Flink native runs 5 times faster than beam: 150,000 strings per second in flink comparing to 30,000 in beam.
When using Avro and schema registry the difference is even more significant - flink native runs 30-80 times faster than beam.
Attached is the java code of both string-to-string pipelines.
Imported from Jira [BEAM-14438](https://issues.apache.org/jira/browse/BEAM-14438). Original Jira may contain additional context.
Reported by: iafek.
Contributor guide
Assessment
This issue has not been assessed yet.