apache / apache/beam

Too many open files due grpc channel not being closed (direct runnner Pubsub source)

Open
#19,260 0 comments 0 reactions 0 assignees View on GitHub
bug direct P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

When '_read_from_pubsub' (runnners/direct/transform_evaluator.py:448) is being invoked, it will create a Pubsub SubscriberClient. This will create a grpc channel which is not being closed. When you run a pipeline with the direct runner which has a pubsub source in it, it will run eventually into an error: "too many open files". This is because these grpc channels are never being closed.

The suggestion is to wrap the code after the creation of the client in a try/finally block and close the channel (exposed on sub_client.api.transport._channel) in the finally block.

 

Some extra info about the issue: https://github.com/googleapis/google-cloud-python/issues/5523

Imported from Jira [BEAM-6708](https://issues.apache.org/jira/browse/BEAM-6708). Original Jira may contain additional context.
Reported by: Boenders.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.