PubSub subscriptions automatically created in Python DirectRunner not reliably cleaned up
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
In the Python DirectRunner, we attempt to garbage-collect automatically-created PubSub subscriptions here (https://github.com/apache/beam/blob/446f311746b74bdfb420ba71d76097d683fde9a1/sdks/python/apache_beam/runners/direct/transform_evaluator.py#L370). However, this mechanism may not succeed, since it is called on the __del__ method when other references may have already been garbage-collected. We should further investigate the mechanism here and attempt to make it more reliable.
Imported from Jira [BEAM-2988](https://issues.apache.org/jira/browse/BEAM-2988). Original Jira may contain additional context.
Reported by: ccy.
Contributor guide
Research direction
Start in sdks/python/apache_beam/runners/direct/transform_evaluator.py around line 370 and inspect how automatically-created PubSub subscriptions are garbage-collected from __del__. Review the imported Jira issue for context, then investigate a cleanup mechanism that remains reliable when other references have already been collected; done means these subscriptions are consistently cleaned up.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100