Duplicate PubSub subscriptions with python direct runner in Jupyter/Colab environment
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
When running a streaming pipeline on Colab with direct runner, ReadFromPubSub can retain old subscriptions and cause message duplication. For example, manually killing a cell that is running a streaming pubsub pipeline does not delete the pubsub subscription. If the cell is rerun, the ReadFromPubSub component will actually be subscribed twice which results in duplicate messages.
Manually deleting old subscriptions (e.g. via the GCP dashboard) temporarily fixes the problem.
This Colab notebook: [https://gist.github.com/hgarrereyn/64ce87cbbcbe9c34ccdd13eafe49e3fb](https://gist.github.com/hgarrereyn/64ce87cbbcbe9c34ccdd13eafe49e3fb) contains a runnable example of the bug.
Imported from Jira [BEAM-10192](https://issues.apache.org/jira/browse/BEAM-10192). Original Jira may contain additional context.
Reported by: hgarrereyn.
Contributor guide
Assessment
This issue has not been assessed yet.