Coder copy overhead
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
More context can be found in discussion here:
[http://mail-archives.apache.org/mod_mbox/beam-dev/201904.mbox/%3CCAOUjMkyKV8npYJfS_PF3Gzo=vwOmB2FRzUtE81ZsrxnM13Tisw@mail.gmail.com%3E](http://mail-archives.apache.org/mod_mbox/beam-dev/201904.mbox/%3CCAOUjMkyKV8npYJfS_PF3Gzo=vwOmB2FRzUtE81ZsrxnM13Tisw@mail.gmail.com%3E)
I am not sure how much is this runner dependent, but each operator's user function receives a copy of data element for isolation. Beam coders does copy by serializing to bytes and then deserialize back. This seems to impact performance and grows with job complexity.
On a simple test pipeline described in discussion thread above, I noticed almost 2x speedup when CoderUtils.copy() just returned the object.
Native Flink job does copy too, but via Kryo, which seems to be doing deep copy more effectively, on object level.
What can be done in Beam to reduce this overhead?
Imported from Jira [BEAM-7206](https://issues.apache.org/jira/browse/BEAM-7206). Original Jira may contain additional context.
Reported by: JozoVilcek.
Contributor guide
Assessment
This issue has not been assessed yet.