_test_multiprocessing._kill_process() uses a fixed 10 s alarm, failing on slow builds
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 35.9k
- PR merge metrics
- PR metrics pending
Description
Bug report
Bug description:
_test_multiprocessing._kill_process() guards the join() of the killed child with a SIGALRM fixed at 10 seconds:
if hasattr(signal, 'alarm'):
# On the Gentoo buildbot waitpid() often seems to block forever.
# We use alarm() to interrupt it if it blocks for too long.
def handler(*args):
raise RuntimeError('join took too long: %s' % p)
old_handler = signal.signal(signal.SIGALRM, handler)
try:
signal.alarm(10)
self.assertEqual(join(), None)
The alarm is meant to turn a blocked waitpid() into a readable error instead of a hang. But the value has been a literal 10 since it was added in 2013 (cc5c728513a), so on a build slow enough that reaping the child legitimately takes longer than ten seconds, it fires on a healthy run and fails the test.
That happened on the "Sanitizers / UBSan" job of 57594aae5e: https://github.com/python/cpython/actions/runs/34016012199/job/101439811595
ERROR: test_interrupt (test.test_multiprocessing_fork.test_processes.WithProcessesTestProcess.test_interrupt)
File ".../Lib/test/_test_multiprocessing.py", line 651, in test_interrupt
exitcode = self._kill_process(multiprocessing.Process.interrupt)
File ".../Lib/test/_test_multiprocessing.py", line 632, in _kill_process
self.assertEqual(join(), None)
File ".../Lib/multiprocessing/popen_fork.py", line 28, in poll
pid, sts = os.waitpid(self.pid, flag)
File ".../Lib/test/_test_multiprocessing.py", line 628, in handler
raise RuntimeError('join took too long: %s' % p)
RuntimeError: join took too long: <Process name='Process-162' pid=18960 parent=17748 started daemon>
The traceback shows the alarm interrupting os.waitpid(), which is the normal path, not a hang.
test.support already provides timeouts for this, and regrtest scales them for slow builds (Lib/test/libregrtest/setup.py raises SHORT_TIMEOUT and LONG_TIMEOUT from --timeout). LONG_TIMEOUT is documented for exactly this use:
# Timeout in seconds to detect when a test hangs.
#
# It is long enough to reduce the risk of test failure on the slowest Python
# buildbots. It should not be used to mark a test as failed if the test takes
# "too long".
and the comment on SHORT_TIMEOUT says "If a test using SHORT_TIMEOUT starts to fail randomly on slow buildbots, use LONG_TIMEOUT instead". The same _kill_process() already uses support.SHORT_TIMEOUT a few lines above, for the event wait.
Reproducer
The alarm fires whenever the child takes longer than ten seconds to be reaped. Replicating the alarm block with a child that is slow to exit:
import multiprocessing, signal, sys, time
def child(event, delay):
def slow_exit(*args):
time.sleep(delay) # a loaded machine: the child is slow to die
sys.exit(0)
signal.signal(signal.SIGINT, slow_exit)
event.set()
time.sleep(100)
if __name__ == '__main__':
alarm_secs, delay = int(sys.argv[1]), float(sys.argv[2])
event = multiprocessing.Event()
p = multiprocessing.Process(target=child, args=(event, delay))
p.daemon = True
p.start()
event.wait(30)
p.interrupt()
def handler(*args):
raise RuntimeError('join took too long: %s' % p)
signal.signal(signal.SIGALRM, handler)
try:
signal.alarm(alarm_secs)
p.join()
print("join returned, exitcode", p.exitcode)
except RuntimeError as exc:
print("RuntimeError:", exc)
finally:
signal.alarm(0)
$ ./python mp_alarm.py 10 12
RuntimeError: join took too long: <Process name='Process-1' pid=13882 parent=13880 started daemon>
$ ./python mp_alarm.py 300 12
join returned, exitcode 0
The four tests that go through _kill_process() (test_interrupt, test_interrupt_no_handler, test_terminate, test_kill) are affected, in every start-method variant.
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux (CI), macOS (reproducer)
Linked PRs
- gh-157185
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in Lib/test/_test_multiprocessing.py at _kill_process(), then compare its alarm handling with support timeout definitions and scaling in Lib/test/libregrtest/setup.py. Check the four affected tests using each start method; done means slow, healthy child reaping no longer triggers the timeout while genuinely blocked joins still fail clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100