apache / apache/beam

SerializablePipelineOptions should not call FileSystems.setDefaultPipelineOptions.

Open
#18,430 0 comments 0 reactions 0 assignees View on GitHub
apex bug core flink P3 runners spark
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

https://github.com/apache/beam/pull/3654 introduces SerializablePipelineOptions, which on deserialization calls FileSystems.setDefaultPipelineOptions.

This is obviously problematic and racy in case the same process uses SerializablePipelineOptions with different filesystem-related options in them.

The reason the PR does this is, Flink and Apex runners were already doing it in their respective SerializablePipelineOptions-like classes (being removed in the PR); and Spark wasn't but probably should have.

I believe this is done for the sake of having the proper filesystem options automatically available on workers in all places where any kind of PipelineOptions are used. Instead, all 3 runners should pick a better place to initialize their workers, and explicitly call FileSystems.setDefaultPipelineOptions there.

It would be even better if FileSystems.setDefaultPipelineOptions didn't exist at all, but that's a topic for a separate JIRA.

CC'ing runner contributors [~aljoscha] [~aviemzur] [~thw]

Imported from Jira [BEAM-2712](https://issues.apache.org/jira/browse/BEAM-2712). Original Jira may contain additional context.
Reported by: jkff.

Contributor guide

Open the contributing guide

Research direction

Start with SerializablePipelineOptions and its deserialization behavior, then inspect the Flink, Apex, and Spark runner code that initializes worker filesystem options. The change is complete when deserialization no longer globally calls FileSystems.setDefaultPipelineOptions and each affected runner explicitly initializes its workers instead.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.