apache / apache/beam

PubSub subscriptions automatically created in Python DirectRunner not reliably cleaned up

Open
#18,518 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

In the Python DirectRunner, we attempt to garbage-collect automatically-created PubSub subscriptions here (https://github.com/apache/beam/blob/446f311746b74bdfb420ba71d76097d683fde9a1/sdks/python/apache_beam/runners/direct/transform_evaluator.py#L370). However, this mechanism may not succeed, since it is called on the __del__ method when other references may have already been garbage-collected. We should further investigate the mechanism here and attempt to make it more reliable.

Imported from Jira [BEAM-2988](https://issues.apache.org/jira/browse/BEAM-2988). Original Jira may contain additional context.
Reported by: ccy.

Contributor guide

Open the contributing guide

Research direction

Start in sdks/python/apache_beam/runners/direct/transform_evaluator.py around line 370 and inspect how automatically-created PubSub subscriptions are garbage-collected from __del__. Review the imported Jira issue for context, then investigate a cleanup mechanism that remains reliable when other references have already been collected; done means these subscriptions are consistently cleaned up.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.