dropbox / dropbox/pyannotate

Hang in subprocess.Popen()

Open
#67 8 comments 5 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.4k
Forks
60
PR merge metrics
No merged PRs in 30d

Description

**[UPDATE: If you have this problem, the solution is to pass `close_fds=True` to `Popen()`]**

I've got a use case where a process being traced for type collection uses `subprocess.Popen()` to execute some helper program, and the `Popen()` call hangs at the line
```py
data = _eintr_retry_call(os.read, errpipe_read, 1048576)
```
in `Popen._execute_child()`. (This is Python 2.7 on Mac, i.e. POSIX.)

That pipe has `FD_CLOEXEC`, so the child is not hitting the `exec()`. Presumably this is because it hangs in a `Queue.put()` operation in ` _trace_dispatch()` (e.g. [here](https://github.com/dropbox/pyannotate/blob/master/pyannotate_runtime/collect_types.py#L853)).

I can think of a gross fix that monkey-patches `os.fork` to disable the profiling hook around the `fork()` so the child doesn't do this. But perhaps there's a more elegant solution (without using `os.register_at_fork()`, which is Python 3.7+ only)? Or the tracing hook could check the pid?

**[UPDATE:]** I can't repro this in a small test program. But it's real, and the `os.fork` monkey-patch fixes it. Not sure what to do about it yet, the monkey-patch seems risky.

Contributor guide

Open the contributing guide

Research direction

Start by reading pyannotate_runtime/collect_types.py around the Queue.put() call referenced in the report, then inspect Python 2.7 POSIX subprocess.Popen() behavior at Popen._execute_child(). Try to reproduce the hang with tracing enabled and compare it with passing close_fds=True; done means establishing a reproducible failure and an agreed safe fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.