[Task] Introduce 1-gpu vs 2-gpu pytest markers

Description

We are getting resource constrained on how many multi-GPU tests we can run. To address this, we should split apart which tests require being run on 2 GPUs and which can run on a single.

The approach is:

• Create pytest markers (registering them in pytest.ini to avoid warnings): multigpu and singlegpu
• Create new github action workflows to execute on the 2GPU runners, and use the multigpu test fixture.
• Run the rest of the tests on the 1GPU runners

This will allow us to run the single and multi-gpu tests in parallel, and only use the limited multi-gpu resources for tests that actually require it.

Annotating tests

To run tests only in multi-gpu settings:

@pytest.mark.multigpu
def test_multi():
    assert True

To run tests in both single- and multiple-gpu settings:

@pytest.mark.singlegpu
@pytest.mark.multigpu
def test_both():
    assert True

To run tests in single-gpu settings only, no annotation is needed:

def test_single():
    assert True

Running tests

To execute the tests, use the following pytest commands:

To run multi-gpu tests:

pytest -m "multigpu"

To run single-gpu tests, we need two commands. One will run the unannotated tests (most of them) the other will run the single-gpu setting of the ones marked as both singlegpu and multigpu

pytest -m "not multigpu"
pytest  -m "singlegpu"

Repositories

• Merlin @nv-alaiacano https://github.com/NVIDIA-Merlin/Merlin/pull/999
• Models @gabrielspmoreira
• NVTabular @karlhigley
• Transformers4Rec @rnyak
• Dataloader @edknv https://github.com/NVIDIA-Merlin/dataloader/pull/151

Start with pytest.ini and the repository's existing GitHub Actions workflows, then identify the test locations that need multigpu or singlegpu markers. Done means the markers are registered, 1GPU and 2GPU workflows run the specified pytest selections in parallel, and the repository's tests are correctly annotated.