pytest-dev / pytest-dev/pytest-xdist

Tests fail on slower computers

Open
#148 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

While trying to validate pytest-xdist works fine on MIPS, tests fail to pass due to too hasty timeout bailing on a single core yeeloong loongson2f machine which I believe to be very roughly matching a 800MHz x86 or worse.

======================================== short test summary info ========================================
SKIP [1] /usr/lib/python2.7/site-packages/_pytest/config.py:1194: no 'gspecs' option found
SKIP [1] testing/test_remote.py:185: skip at module level illegal in pytest 3.0
FAIL testing/acceptance_test.py::TestDistribution::()::test_keyboard_interrupt_dist
FAIL testing/test_looponfail.py::TestFunctional::()::test_fail_to_ok
XFAIL testing/acceptance_test.py::TestDistEach::()::test_simple_diffoutput
  reason: [NOTRUN] other python versions might not have py.test installed
XFAIL testing/acceptance_test.py::test_terminate_on_hangingnode
XFAIL testing/acceptance_test.py::test_session_hooks
  reason: [NOTRUN] works if run outside test suite
XFAIL testing/test_boxed.py::test_functional_boxed_capturing[sys]
  capture cleanup needed
XFAIL testing/test_boxed.py::test_functional_boxed_capturing[fd]
  capture cleanup needed
XFAIL testing/test_dsession.py::TestDistReporter::()::test_rsync_printing
XFAIL testing/test_dsession.py::test_pytest_issue419
  duplicate test ids not supported yet
XFAIL testing/test_remote.py::test_remoteinitconfig
  #59
XFAIL testing/test_remote.py::TestSlaveInteractor::()::test_happy_run_events_converted
  reason: implement a simple test for event production
XFAIL testing/test_slavemanage.py::TestNodeManager::()::test_rsync_roots_no_roots
  reason: [NOTRUN] 
XPASS testing/acceptance_test.py::TestNodeFailure::()::test_each_multiple #20: xdist race condition on node restart

=============================================== FAILURES ================================================
_____________________________ TestDistribution.test_keyboard_interrupt_dist _____________________________

self = <acceptance_test.TestDistribution instance at 0x7381f9e0>
testdir = <Testdir local('/var/tmp/portage/dev-python/pytest-xdist-1.15.0/temp/pytest-of-portage/pytest-0/testdir/test_keyboard_interrupt_dist0')>

    def test_keyboard_interrupt_dist(self, testdir):
        # xxx could be refined to check for return code
        testdir.makepyfile("""
                def test_sleep():
                    import time
                    time.sleep(10)
            """)
        child = testdir.spawn_pytest("-n1 -v")
>       child.expect(".*test_sleep.*")

../../../../../work/pytest-xdist-1.15.0/testing/acceptance_test.py:246: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python2.7/site-packages/pexpect/spawnbase.py:321: in expect
    timeout, searchwindowsize, async)
/usr/lib/python2.7/site-packages/pexpect/spawnbase.py:345: in expect_list
    return exp.expect_loop(timeout)
/usr/lib/python2.7/site-packages/pexpect/expect.py:107: in expect_loop
    return self.timeout(e)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pexpect.expect.Expecter object at 0x75ece110>, err = TIMEOUT('Timeout exceeded.',)

    def timeout(self, err=None):
        spawn = self.spawn
        from . import TIMEOUT
    
        spawn.before = spawn.buffer
        spawn.after = TIMEOUT
        index = self.searcher.timeout_index
        if index >= 0:
            spawn.match = TIMEOUT
            spawn.match_index = index
            return index
        else:
            spawn.match = None
            spawn.match_index = None
            msg = str(spawn)
            msg += '\nsearcher: %s' % self.searcher
            if err is not None:
                msg = str(err) + '\n' + msg
>           raise TIMEOUT(msg)
E           TIMEOUT: Timeout exceeded.
E           <pexpect.pty_spawn.spawn object at 0x73836e10>
E           command: /usr/bin/python2.7
E           args: ['/usr/bin/python2.7', '/usr/lib/python2.7/site-packages/pytest.py', '--basetemp=/var/tmp/portage/dev-python/pytest-xdist-1.15.0/temp/pytest-of-portage/pytest-0/testdir/test_keyboard_interrupt_dist0/pexpect', '-n1', '-v']
E           buffer (last 100 chars): 'ist-1.15.0/temp/pytest-of-portage/pytest-0/testdir/test_keyboard_interrupt_dist0\r\n\x1b[0m\x1b[1m\rgw0 C\x1b[0m'
E           before (last 100 chars): 'ist-1.15.0/temp/pytest-of-portage/pytest-0/testdir/test_keyboard_interrupt_dist0\r\n\x1b[0m\x1b[1m\rgw0 C\x1b[0m'
E           after: <class 'pexpect.exceptions.TIMEOUT'>
E           match: None
E           match_index: None
E           exitstatus: None
E           flag_eof: False
E           pid: 21194
E           child_fd: 13
E           closed: False
E           timeout: 10.0
E           delimiter: <class 'pexpect.exceptions.EOF'>
E           logfile: <open file '/var/tmp/portage/dev-python/pytest-xdist-1.15.0/temp/pytest-of-portage/pytest-0/testdir/test_keyboard_interrupt_dist0/spawn.out', mode 'wb' at 0x738a35a0>
E           logfile_read: None
E           logfile_send: None
E           maxread: 2000
E           ignorecase: False
E           searchwindowsize: None
E           delaybeforesend: 0.05
E           delayafterclose: 0.1
E           delayafterterminate: 0.1
E           searcher: searcher_re:
E               0: re.compile(".*test_sleep.*")

/usr/lib/python2.7/site-packages/pexpect/expect.py:70: TIMEOUT
____________________________________ TestFunctional.test_fail_to_ok _____________________________________

self = <test_looponfail.TestFunctional instance at 0x7585cf58>
testdir = <Testdir local('/var/tmp/portage/dev-python/pytest-xdist-1.15.0/temp/pytest-of-portage/pytest-0/testdir/test_fail_to_ok0')>

    def test_fail_to_ok(self, testdir):
        p = testdir.makepyfile("""
                def test_one():
                    x = 0
                    assert x == 1
            """)
        # p = testdir.mkdir("sub").join(p1.basename)
        # p1.move(p)
        child = testdir.spawn_pytest("-f %s --traceconfig" % p)
>       child.expect("def test_one")

../../../../../work/pytest-xdist-1.15.0/testing/test_looponfail.py:249: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python2.7/site-packages/pexpect/spawnbase.py:321: in expect
    timeout, searchwindowsize, async)
/usr/lib/python2.7/site-packages/pexpect/spawnbase.py:345: in expect_list
    return exp.expect_loop(timeout)
/usr/lib/python2.7/site-packages/pexpect/expect.py:107: in expect_loop
    return self.timeout(e)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pexpect.expect.Expecter object at 0x7585f030>, err = TIMEOUT('Timeout exceeded.',)

    def timeout(self, err=None):
        spawn = self.spawn
        from . import TIMEOUT
    
        spawn.before = spawn.buffer
        spawn.after = TIMEOUT
        index = self.searcher.timeout_index
        if index >= 0:
            spawn.match = TIMEOUT
            spawn.match_index = index
            return index
        else:
            spawn.match = None
            spawn.match_index = None
            msg = str(spawn)
            msg += '\nsearcher: %s' % self.searcher
            if err is not None:
                msg = str(err) + '\n' + msg
>           raise TIMEOUT(msg)
E           TIMEOUT: Timeout exceeded.
E           <pexpect.pty_spawn.spawn object at 0x7585f070>
E           command: /usr/bin/python2.7
E           args: ['/usr/bin/python2.7', '/usr/lib/python2.7/site-packages/pytest.py', '--basetemp=/var/tmp/portage/dev-python/pytest-xdist-1.15.0/temp/pytest-of-portage/pytest-0/testdir/test_fail_to_ok0/pexpect', '-f', '/var/tmp/portage/dev-python/pytest-xdist-1.15.0/temp/pytest-of-portage/pytest-0/testdir/test_fail_to_ok0/test_fail_to_ok.py', '--traceconfig']
E           buffer (last 100 chars): ''
E           before (last 100 chars): ''
E           after: <class 'pexpect.exceptions.TIMEOUT'>
E           match: None
E           match_index: None
E           exitstatus: None
E           flag_eof: False
E           pid: 21630
E           child_fd: 13
E           closed: False
E           timeout: 10.0
E           delimiter: <class 'pexpect.exceptions.EOF'>
E           logfile: <open file '/var/tmp/portage/dev-python/pytest-xdist-1.15.0/temp/pytest-of-portage/pytest-0/testdir/test_fail_to_ok0/spawn.out', mode 'wb' at 0x758633e8>
E           logfile_read: None
E           logfile_send: None
E           maxread: 2000
E           ignorecase: False
E           searchwindowsize: None
E           delaybeforesend: 0.05
E           delayafterclose: 0.1
E           delayafterterminate: 0.1
E           searcher: searcher_re:
E               0: re.compile("def test_one")

/usr/lib/python2.7/site-packages/pexpect/expect.py:70: TIMEOUT
=============== 2 failed, 94 passed, 2 skipped, 10 xfailed, 1 xpassed in 1077.48 seconds ================

system type		: lemote-yeeloong-2f-8.9inches
machine			: Unknown
processor		: 0
cpu model		: ICT Loongson-2 V0.3  FPU V0.1
BogoMIPS		: 528.38
wait instruction	: yes
microsecond timers	: yes
tlb_entries		: 64
extra interrupt vector	: no
hardware watchpoint	: yes, count: 0, address/irw mask: []
ASEs implemented	:
shadow register sets	: 1
kscratch registers	: 0
core			: 0
VCED exceptions		: not available
VCEI exceptions		: not available

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with testing/acceptance_test.py::TestDistribution::test_keyboard_interrupt_dist and testing/test_looponfail.py::TestFunctional::test_fail_to_ok, then inspect their pexpect waits and the reported 10-second timeout. Run these tests on a slow single-core system and adjust the timeout behavior so both tests complete reliably without premature failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.