Hang in subprocess.Popen()
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 60
- PR merge metrics
- No merged PRs in 30d
Description
**[UPDATE: If you have this problem, the solution is to pass `close_fds=True` to `Popen()`]**
I've got a use case where a process being traced for type collection uses `subprocess.Popen()` to execute some helper program, and the `Popen()` call hangs at the line
```py
data = _eintr_retry_call(os.read, errpipe_read, 1048576)
```
in `Popen._execute_child()`. (This is Python 2.7 on Mac, i.e. POSIX.)
That pipe has `FD_CLOEXEC`, so the child is not hitting the `exec()`. Presumably this is because it hangs in a `Queue.put()` operation in ` _trace_dispatch()` (e.g. [here](https://github.com/dropbox/pyannotate/blob/master/pyannotate_runtime/collect_types.py#L853)).
I can think of a gross fix that monkey-patches `os.fork` to disable the profiling hook around the `fork()` so the child doesn't do this. But perhaps there's a more elegant solution (without using `os.register_at_fork()`, which is Python 3.7+ only)? Or the tracing hook could check the pid?
**[UPDATE:]** I can't repro this in a small test program. But it's real, and the `os.fork` monkey-patch fixes it. Not sure what to do about it yet, the monkey-patch seems risky.
Contributor guide
Research direction
Start by reading pyannotate_runtime/collect_types.py around the Queue.put() call referenced in the report, then inspect Python 2.7 POSIX subprocess.Popen() behavior at Popen._execute_child(). Try to reproduce the hang with tracing enabled and compare it with passing close_fds=True; done means establishing a reproducible failure and an agreed safe fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devtools
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100