Reuse job servers when possible in (spark|flink)_runner.py.
Open
core
flink
improvement
P3
python
runners
spark
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
If multiple Beam pipelines are run on Spark or Flink from the same Python script, currently a new Java job server will be created for each pipeline. This can waste lots of resources (i.e. memory), since job servers are often functionally equivalent and therefore reusable.
Imported from Jira [BEAM-12439](https://issues.apache.org/jira/browse/BEAM-12439). Original Jira may contain additional context.
Reported by: ibzib.
Contributor guide
Assessment
This issue has not been assessed yet.