apache / apache/beam

Make the spark runner not serialize data unless spark is spilling to disk

Open
#19,228 0 comments 0 reactions 0 assignees View on GitHub
improvement P3 runners spark
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Currently for storage level MEMORY_ONLY, Beam does not coder-ify the data. This lets Spark keep the data in memory avoiding the serialization round trip. Unfortunately the logic is fairly coarse - as soon as you switch to MEMORY_AND_DISK, Beam coder-ifys the data even though Spark might have chosen to keep the data in memory, incurring the serialization overhead.

 

Ideally Beam would serialize the data lazily - as Spark chooses to spill to disk. This would be a change in behavior when using beam, but luckily Spark has a solution for folks that want data serialized in memory - MEMORY_AND_DISK_SER will keep the data serialized.

Imported from Jira [BEAM-5775](https://issues.apache.org/jira/browse/BEAM-5775). Original Jira may contain additional context.
Reported by: mikekap.

Contributor guide

Open the contributing guide

Research direction

Start by locating the Spark runner logic that handles MEMORY_ONLY and MEMORY_AND_DISK storage levels, then trace where Beam coder-ifies cached data. Confirm the behavior against Spark's spilling semantics and verify that serialization is deferred until data is spilled, while MEMORY_AND_DISK_SER remains the explicit serialized-in-memory option.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.