pytest-dev / pytest-dev/pytest-xdist

Constantly hanging test run with the plugin

Open
#872 14 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
288
Avg merge
9h 30m
Merged PRs (30d)
2

Description

Hello,

We are using:

platform linux -- Python 3.9.5, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
plugins: forked-1.4.0, xdist-2.5.0, pytest_check-1.0.4, teamcity-messages-1.29, anyio-3.3.4, testrail-2.9.1, dependency-0.5.1

When trying to execute pytest using xdist on remote windows host by loadfile get hanging test run.

The command:

python3 -m pytest -vv --dist=loadfile --tx ssh=admin@test-host-ip --rsyncdir /tmp/autotests_rsync C:\\users\\admin\\pyexecnetcache\\autotests_rsync\\autotests\\testsuite\\positive

The hanging appears in test, which using waiter to get value from postgresql db via SQLAlchemy ORM.

We're passing value from test suite to the following test:

start_time = Waiter.wait_new(lambda: DbTestData.get_session_records_column_by_record_id(
            DbTestData.start_time, record_id)[0][0],
                                     check_func=CheckFunctions.check_none,
                                     error_message=f"Error")
assert start_time is not None, f"Record start_time in db = {start_time}, expected not None"
   def query(*args):
        session = SessionHolder.get_session()
        result = session.query(*args)
        session.commit()
        return result

which using this waiter:

    @staticmethod
    def wait_new(func: Callable, check_func: Callable = CheckFunctions.check_empty, timeout_value: int = 20,
                 timeout_interval: int = 1, error_message: str = ""):
        print(f"Func = {func}")
        value = waiter_exception
        exc_raise_if_fail = TestWaiterException()
        timeout = 0
        in_while = True
        Logger.utils_logger.debug(f"timeout_value = {timeout_value}, timeout_interval = {timeout_interval})")
        while in_while:
            print(f"in_while loop")
            try:
                print(f"Trying execute func")
                value = func()
            except Exception as ex:
                print(f"Exception")
                if timeout == timeout_value:
                    exc_raise_if_fail.with_traceback(sys.exc_info()[2])
                    exc_raise_if_fail.txt += ": " + ex.args[0]
                    in_while = False
                value = waiter_exception
                Logger.utils_logger.debug(f"Exception", exc_info=True)
            finally:
                print(f"Finally")
                Logger.utils_logger.debug(f"Current value: {value}")
                if (timeout > timeout_value) or (value != waiter_exception and not check_func(value)):
                    print(f"Break")
                    break
                else:
                    print(f"Else")
                    timeout += timeout_interval
                    time.sleep(timeout_interval)
        if value == waiter_exception:
            Logger.utils_logger.critical(f"{exc_raise_if_fail.txt}, {error_message}")
            raise exc_raise_if_fail
        return value

It just hangs permanently while executing waiter only when we using xdist plugin.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the command using xdist's remote Windows host and loadfile distribution, then inspect the Waiter.wait_new call and SessionHolder.get_session query path shown in the report. Compare the hanging behavior with and without xdist; done means identifying the cause of the permanent wait and documenting or testing the resulting fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.