SerializablePipelineOptions should not call FileSystems.setDefaultPipelineOptions.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
https://github.com/apache/beam/pull/3654 introduces SerializablePipelineOptions, which on deserialization calls FileSystems.setDefaultPipelineOptions.
This is obviously problematic and racy in case the same process uses SerializablePipelineOptions with different filesystem-related options in them.
The reason the PR does this is, Flink and Apex runners were already doing it in their respective SerializablePipelineOptions-like classes (being removed in the PR); and Spark wasn't but probably should have.
I believe this is done for the sake of having the proper filesystem options automatically available on workers in all places where any kind of PipelineOptions are used. Instead, all 3 runners should pick a better place to initialize their workers, and explicitly call FileSystems.setDefaultPipelineOptions there.
It would be even better if FileSystems.setDefaultPipelineOptions didn't exist at all, but that's a topic for a separate JIRA.
CC'ing runner contributors [~aljoscha] [~aviemzur] [~thw]
Imported from Jira [BEAM-2712](https://issues.apache.org/jira/browse/BEAM-2712). Original Jira may contain additional context.
Reported by: jkff.
Contributor guide
Research direction
Start with SerializablePipelineOptions and its deserialization behavior, then inspect the Flink, Apex, and Spark runner code that initializes worker filesystem options. The change is complete when deserialization no longer globally calls FileSystems.setDefaultPipelineOptions and each affected runner explicitly initializes its workers instead.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100