Make the spark runner not serialize data unless spark is spilling to disk
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Currently for storage level MEMORY_ONLY, Beam does not coder-ify the data. This lets Spark keep the data in memory avoiding the serialization round trip. Unfortunately the logic is fairly coarse - as soon as you switch to MEMORY_AND_DISK, Beam coder-ifys the data even though Spark might have chosen to keep the data in memory, incurring the serialization overhead.
Ideally Beam would serialize the data lazily - as Spark chooses to spill to disk. This would be a change in behavior when using beam, but luckily Spark has a solution for folks that want data serialized in memory - MEMORY_AND_DISK_SER will keep the data serialized.
Imported from Jira [BEAM-5775](https://issues.apache.org/jira/browse/BEAM-5775). Original Jira may contain additional context.
Reported by: mikekap.
Contributor guide
Research direction
Start by locating the Spark runner logic that handles MEMORY_ONLY and MEMORY_AND_DISK storage levels, then trace where Beam coder-ifies cached data. Confirm the behavior against Spark's spilling semantics and verify that serialization is deferred until data is spilled, while MEMORY_AND_DISK_SER remains the explicit serialized-in-memory option.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100