apache / apache/beam

[Bug]: Python SDK: Dependency installation hangs sometimes

Open
#24,357 17 comments 0 reactions 0 assignees View on GitHub
bug harness P2 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

Disclaimer: This bug reports refers to the current master branch, which contains major changes to the container boot.go, but it's very likely that it applies to previous versions as well, since the parts which most likely trigger the bug were not changed.

The Python SDK boot.go binary installs dependencies from a setup.py automatically here: https://github.com/apache/beam/blob/09606de8779ca284666581c7c2da314df4f9ddb1/sdks/python/container/piputil.go#L90-L91

Most of the time, it works fine, but sometimes the `pip` process seems to hang for no reason. The problem can be resolved by attaching to the container running the SDK and reading stdout of the `boot` worker process:

```
$ kubectl exec -ti pod/beam-flinker-cluster-taskmanager -c beam-worker-pool -- bash
# ps aux | grep '/opt/apache/beam/boot --id=1-1'
# cat /proc//fd/1
^C
# exit
```

I suppose this issue occurs due to a stdout buffer fillup. The boot binary uses the following utility for running `pip`, which redirects `pip`'s stdout and stderr to `os.stdout` and `os.stderr`:
https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/execx/exec.go#L34-L36

A solution would perhaps be to either not redirect stdout or to write it directly to `/dev/null`.

### Issue Priority

Priority: 2

### Issue Component

Component: sdk-py-harness

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.