python / python/cpython

_test_multiprocessing._kill_process() uses a fixed 10 s alarm, failing on slow builds

Open
#157,184 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

tests topic-multiprocessing type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:

_test_multiprocessing._kill_process() guards the join() of the killed child with a SIGALRM fixed at 10 seconds:

        if hasattr(signal, 'alarm'):
            # On the Gentoo buildbot waitpid() often seems to block forever.
            # We use alarm() to interrupt it if it blocks for too long.
            def handler(*args):
                raise RuntimeError('join took too long: %s' % p)
            old_handler = signal.signal(signal.SIGALRM, handler)
            try:
                signal.alarm(10)
                self.assertEqual(join(), None)

The alarm is meant to turn a blocked waitpid() into a readable error instead of a hang. But the value has been a literal 10 since it was added in 2013 (cc5c728513a), so on a build slow enough that reaping the child legitimately takes longer than ten seconds, it fires on a healthy run and fails the test.

That happened on the "Sanitizers / UBSan" job of 57594aae5e: https://github.com/python/cpython/actions/runs/34016012199/job/101439811595

ERROR: test_interrupt (test.test_multiprocessing_fork.test_processes.WithProcessesTestProcess.test_interrupt)
  File ".../Lib/test/_test_multiprocessing.py", line 651, in test_interrupt
    exitcode = self._kill_process(multiprocessing.Process.interrupt)
  File ".../Lib/test/_test_multiprocessing.py", line 632, in _kill_process
    self.assertEqual(join(), None)
  File ".../Lib/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
  File ".../Lib/test/_test_multiprocessing.py", line 628, in handler
    raise RuntimeError('join took too long: %s' % p)
RuntimeError: join took too long: <Process name='Process-162' pid=18960 parent=17748 started daemon>

The traceback shows the alarm interrupting os.waitpid(), which is the normal path, not a hang.

test.support already provides timeouts for this, and regrtest scales them for slow builds (Lib/test/libregrtest/setup.py raises SHORT_TIMEOUT and LONG_TIMEOUT from --timeout). LONG_TIMEOUT is documented for exactly this use:

# Timeout in seconds to detect when a test hangs.
#
# It is long enough to reduce the risk of test failure on the slowest Python
# buildbots. It should not be used to mark a test as failed if the test takes
# "too long".

and the comment on SHORT_TIMEOUT says "If a test using SHORT_TIMEOUT starts to fail randomly on slow buildbots, use LONG_TIMEOUT instead". The same _kill_process() already uses support.SHORT_TIMEOUT a few lines above, for the event wait.

Reproducer

The alarm fires whenever the child takes longer than ten seconds to be reaped. Replicating the alarm block with a child that is slow to exit:

import multiprocessing, signal, sys, time

def child(event, delay):
    def slow_exit(*args):
        time.sleep(delay)      # a loaded machine: the child is slow to die
        sys.exit(0)
    signal.signal(signal.SIGINT, slow_exit)
    event.set()
    time.sleep(100)

if __name__ == '__main__':
    alarm_secs, delay = int(sys.argv[1]), float(sys.argv[2])
    event = multiprocessing.Event()
    p = multiprocessing.Process(target=child, args=(event, delay))
    p.daemon = True
    p.start()
    event.wait(30)
    p.interrupt()

    def handler(*args):
        raise RuntimeError('join took too long: %s' % p)
    signal.signal(signal.SIGALRM, handler)
    try:
        signal.alarm(alarm_secs)
        p.join()
        print("join returned, exitcode", p.exitcode)
    except RuntimeError as exc:
        print("RuntimeError:", exc)
    finally:
        signal.alarm(0)
$ ./python mp_alarm.py 10 12
RuntimeError: join took too long: <Process name='Process-1' pid=13882 parent=13880 started daemon>
$ ./python mp_alarm.py 300 12
join returned, exitcode 0

The four tests that go through _kill_process() (test_interrupt, test_interrupt_no_handler, test_terminate, test_kill) are affected, in every start-method variant.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux (CI), macOS (reproducer)

Linked PRs
  • gh-157185

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in Lib/test/_test_multiprocessing.py at _kill_process(), then compare its alarm handling with support timeout definitions and scaling in Lib/test/libregrtest/setup.py. Check the four affected tests using each start method; done means slow, healthy child reaping no longer triggers the timeout while genuinely blocked joins still fail clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.