[Bug]: Python SDK: Dependency installation hangs sometimes
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
### What happened?
Disclaimer: This bug reports refers to the current master branch, which contains major changes to the container boot.go, but it's very likely that it applies to previous versions as well, since the parts which most likely trigger the bug were not changed.
The Python SDK boot.go binary installs dependencies from a setup.py automatically here: https://github.com/apache/beam/blob/09606de8779ca284666581c7c2da314df4f9ddb1/sdks/python/container/piputil.go#L90-L91
Most of the time, it works fine, but sometimes the `pip` process seems to hang for no reason. The problem can be resolved by attaching to the container running the SDK and reading stdout of the `boot` worker process:
```
$ kubectl exec -ti pod/beam-flinker-cluster-taskmanager -c beam-worker-pool -- bash
# ps aux | grep '/opt/apache/beam/boot --id=1-1'
# cat /proc//fd/1
^C
# exit
```
I suppose this issue occurs due to a stdout buffer fillup. The boot binary uses the following utility for running `pip`, which redirects `pip`'s stdout and stderr to `os.stdout` and `os.stderr`:
https://github.com/apache/beam/blob/master/sdks/go/pkg/beam/util/execx/exec.go#L34-L36
A solution would perhaps be to either not redirect stdout or to write it directly to `/dev/null`.
### Issue Priority
Priority: 2
### Issue Component
Component: sdk-py-harness
Contributor guide
Assessment
This issue has not been assessed yet.