pytest-dev / pytest-dev/pytest-xdist

pytest-asyncio Hangs near finish using GPT-3.5 inference fixture

Open
#1,084 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.9k
Forks
287
Avg merge
9h 30m
Merged PRs (30d)
2

Description

Issue Description

When running pytest-asyncio with pytest-xdist, my tests consistently hang between 90-100% completion. I suspect there might be an issue with resource acquisition not being properly managed at the end of the test suite.

I am using a fixture for running GPT-3.5 inference.

It accepts --models as an argument and uses it to parameterize to create a test suite.

def pytest_generate_tests(metafunc):
    """Generate tests based on custom command-line options."""
    if "model" in metafunc.fixturenames and metafunc.config.option.models:
        models = metafunc.config.option.models
        if models == "all":
            models = list(ModelType)
        elif models == "served":
            models = list(ServedModelType)

        model_fixtures = []
        ids = []
        for model in models:
            if model not in ModelType:
                raise ValueError(f"Unknown model: {model}")
            model_fixtures.append(model)
            ids.append(model)

        metafunc.parametrize("model", model_fixtures, ids=ids, scope="session")

@pytest.mark.asyncio
async def test(model, agenerate_message):
    response = await agenerate_message(model=model)
    passed = "passed" in response
    assert passed

I added a session-scoped fixture to manage the event loop and ensure clean-up, the problem persists.

@pytest.fixture(scope="session")
def event_loop():
    loop = asyncio.get_event_loop()
    yield loop

    pending = asyncio.tasks.all_tasks(loop)
    loop.run_until_complete(asyncio.gather(*pending))
    loop.run_until_complete(asyncio.sleep(1))
    loop.close()

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the pytest_generate_tests hook and the session-scoped event_loop fixture shown in the report. Reproduce the hang with pytest-asyncio, pytest-xdist, the --models parameter, and the GPT-3.5 inference fixture, then inspect pending tasks during teardown. Done means identifying a reproducible xdist or fixture interaction and documenting or fixing its cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.