Stop job service when pipeline execution finishes
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Currently, job servers are shut down when the Python script exits [1]. A better long-term solution would be to shut them down instead when the pipeline is finished executing, such as [2]. This will put resource management in a common code path that is less error-prone.
[1] [https://github.com/apache/beam/blob/c5f43342f914fc8ff367b86fb9294c38436ed3ce/sdks/python/apache_beam/runners/portability/job_server.py#L73](https://github.com/apache/beam/blob/c5f43342f914fc8ff367b86fb9294c38436ed3ce/sdks/python/apache_beam/runners/portability/job_server.py#L73)
[2] [https://github.com/apache/beam/blob/c5f43342f914fc8ff367b86fb9294c38436ed3ce/sdks/python/apache_beam/runners/portability/portable_runner.py#L451](https://github.com/apache/beam/blob/c5f43342f914fc8ff367b86fb9294c38436ed3ce/sdks/python/apache_beam/runners/portability/portable_runner.py#L451)
Imported from Jira [BEAM-8103](https://issues.apache.org/jira/browse/BEAM-8103). Original Jira may contain additional context.
Reported by: ibzib.
Contributor guide
Assessment
This issue has not been assessed yet.