apache / apache/beam

Beam performance degraded significantly since 2.4

Open
#19,065 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Starting from Beam 2.4 onwards the *InMemoryStateInternals* class in the *beam-runners-core-java* module does an expensive Coder encode/decode operation when copying object state. This has significant impact on performance and pipelines that previously took low minutes do not finish within hours in our case. Based on the discussion on the dev mailing list, the main motivation for this change was to enforce Coder sanity, something that should arguably remain within the realm of the DirectRunner and should not leak into the core layer.

Links to commits that introduced the new behaviour:
* [https://github.com/apache/beam/commit/32a427c](https://github.com/apache/beam/commit/32a427c)
* [https://github.com/apache/beam/commit/8151d82](https://github.com/apache/beam/commit/8151d82)

 

Additional details and surrounding discussion can be found here:
* https://lists.apache.org/thread.html/1e329318a4dafe27b9ff304d9460d05d8966e5ceebaf4ebfb948e2b8@%3Cdev.beam.apache.org%3E

 

Imported from Jira [BEAM-4750](https://issues.apache.org/jira/browse/BEAM-4750). Original Jira may contain additional context.
Reported by: janotav.

Contributor guide

Open the contributing guide

Research direction

Start with InMemoryStateInternals in the beam-runners-core-java module and inspect commits 32a427c and 8151d82 to understand the added Coder encode/decode behavior. Review the linked mailing-list discussion for the intended boundary between core and DirectRunner validation. Done means resolving the regression without losing the required Coder sanity checks and confirming the performance impact is addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.