BUG: Multiple `PyQt`/`PySide` Versions Not Supported

Bug

I ran into this issue when trying to fix a bug with pyvistaqt. It appears, for some reason, when all of

• PyQt5
• PySide2
• PyQt6
• PySide6

are installed, pytest-qt fails to create a QApplication instance, causing tests to silently fail.

NOTE: QT_API is not set to anything on my system when this occurs.

System Information

OS: Windows 10, x64-bit
Python: 3.8.10 x64-bit (CPython)
pytest-qt: 4.1.0

Steps to Reproduce

1. Clone the pyvistaqt repo and cd into the folder

git clone https://github.com/pyvista/pyvistaqt.git
cd pyvistaqt

2. Create and activate a virtual environment

py -m venv .venv && .venv\Scripts\Activate.ps1

3. Install dependencies

py -m pip install --upgrade pip
py -m pip install -r requirements_test.txt
py -m pip install -r requirements_docs.txt

5. First, install only PySide6

py -m pip install pyside6

6. Sanity Check: Run tests to verify everything is working correctly (All Pass)

pytest

7. Now install PyQt5, PySide2, and PyQt6

py -m pip install pyqt5 pyside2 pyqt6

8. Rerun pytest and tests will fail silently

PS> pytest
=================================== test session starts =======================================
platform win32 -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
PySide6 6.3.1 -- Qt runtime 6.3.1 -- Qt compiled 6.3.1
rootdir: %USERPROFILE%\Code\external\pyvistaqt-demo\pyvistaqt, configfile: pytest.ini
plugins: cov-3.0.0, memprof-0.2.0, mypy-plugins-1.9.3, mypy-testing-0.0.11, qt-4.1.0, sphinx-0.4.0
collected 2 items

tests\test_plotting.py

9. Re-run pytest in verbose mode while printing output to stdout/stderr

PS> pytest -v -s

==================================== test session starts ======================================
platform win32 -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0 -- %USERPROFILE%\code\external\pyvistaqt-demo\pyvistaqt\.venv\scripts\python.exe
cachedir: .pytest_cache
PySide6 6.3.1 -- Qt runtime 6.3.1 -- Qt compiled 6.3.1
rootdir: %USERPROFILE%\Code\external\pyvistaqt-demo\pyvistaqt, configfile: pytest.ini
plugins: cov-3.0.0, memprof-0.2.0, mypy-plugins-1.9.3, mypy-testing-0.0.11, qt-4.1.0, sphinx-0.4.0
collected 2 items

tests/test_plotting.py::test_create_menu_bar QWidget: Must construct a QApplication before a QWidget

10. Add an assert to the offending test for details

def test_create_menu_bar(qtbot):
    assert QtWidgets.QApplication.instance() is not None
    menu_bar = _create_menu_bar(parent=None)
    qtbot.addWidget(menu_bar)

11. Rerun test in verbose mode while outputing to stdout/stderr

PS> pytest -v -s
==================================== test session starts ======================================
platform win32 -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0 -- %USERPROFILE%\code\external\pyvistaqt-demo\pyvistaqt\.venv\scripts\python.exe
cachedir: .pytest_cache
PySide6 6.3.1 -- Qt runtime 6.3.1 -- Qt compiled 6.3.1
rootdir: %USERPROFILE%\Code\external\pyvistaqt-demo\pyvistaqt, configfile: pytest.ini
plugins: cov-3.0.0, memprof-0.2.0, mypy-plugins-1.9.3, mypy-testing-0.0.11, qt-4.1.0, sphinx-0.4.0
collected 2 items

tests/test_plotting.py::test_create_menu_bar FAILED
tests/test_qt.py::test_no_qt_binding PASSED

========================================= FAILURES ======================================
____________________________________________________________ test_create_menu_bar _________________________________________________________

qtbot = <pytestqt.qtbot.QtBot object at 0x00000237D940E160>

    def test_create_menu_bar(qtbot):
>       assert QtWidgets.QApplication.instance() is not None
E       AssertionError: assert None is not None
E        +  where None = <built-in function instance>()
E        +    where <built-in function instance> = <class 'PyQt5.QtWidgets.QApplication'>.instance
E        +      where <class 'PyQt5.QtWidgets.QApplication'> = QtWidgets.QApplication

tests\test_plotting.py:79: AssertionError
================================== memory consumption estimates ==============================
tests::test_plotting.py::test_create_menu_bar  - 40.0 KB
================================== short test summary info ====================================
FAILED tests/test_plotting.py::test_create_menu_bar - AssertionError: assert None is not None
================================== 1 failed, 1 passed in 0.79s ===================================

It appears there is a disconnect between the python Qt library registered by pytest:

PySide6 6.3.1 -- Qt runtime 6.3.1 -- Qt compiled 6.3.1

and the one picked up by pytest-qt

>       assert QtWidgets.QApplication.instance() is not None
E       AssertionError: assert None is not None
E        +  where None = <built-in function instance>()
E        +    where <built-in function instance> = <class 'PyQt5.QtWidgets.QApplication'>.instance
E        +      where <class 'PyQt5.QtWidgets.QApplication'> = QtWidgets.QApplication

Expected Behavior

pytest-qt can be run with multiple versions of python Qt libraries installed simultaneously for testing purposes.

Actual Behavior

pytest-qt has a disconnect between the python Qt binding registered with pytest and the one it uses.

Reproduce the failure with the installation steps and compare the binding reported by pytest with the QApplication used by pytest-qt. Use pyvistaqt's tests/test_plotting.py::test_create_menu_bar and tests/test_qt.py::test_no_qt_binding as checks, including the QT_API environment behavior. Done means pytest-qt selects and creates the same Qt binding when all four bindings are installed.