pytest-dev / pytest-dev/pytest-xdist
pytest-asyncio Hangs near finish using GPT-3.5 inference fixture
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 287
- Avg merge
- 9h 30m
- Merged PRs (30d)
- 2
Description
Issue Description
When running pytest-asyncio with pytest-xdist, my tests consistently hang between 90-100% completion. I suspect there might be an issue with resource acquisition not being properly managed at the end of the test suite.
I am using a fixture for running GPT-3.5 inference.
It accepts --models as an argument and uses it to parameterize to create a test suite.
def pytest_generate_tests(metafunc):
"""Generate tests based on custom command-line options."""
if "model" in metafunc.fixturenames and metafunc.config.option.models:
models = metafunc.config.option.models
if models == "all":
models = list(ModelType)
elif models == "served":
models = list(ServedModelType)
model_fixtures = []
ids = []
for model in models:
if model not in ModelType:
raise ValueError(f"Unknown model: {model}")
model_fixtures.append(model)
ids.append(model)
metafunc.parametrize("model", model_fixtures, ids=ids, scope="session")
@pytest.mark.asyncio
async def test(model, agenerate_message):
response = await agenerate_message(model=model)
passed = "passed" in response
assert passed
I added a session-scoped fixture to manage the event loop and ensure clean-up, the problem persists.
@pytest.fixture(scope="session")
def event_loop():
loop = asyncio.get_event_loop()
yield loop
pending = asyncio.tasks.all_tasks(loop)
loop.run_until_complete(asyncio.gather(*pending))
loop.run_until_complete(asyncio.sleep(1))
loop.close()
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the pytest_generate_tests hook and the session-scoped event_loop fixture shown in the report. Reproduce the hang with pytest-asyncio, pytest-xdist, the --models parameter, and the GPT-3.5 inference fixture, then inspect pending tasks during teardown. Done means identifying a reproducible xdist or fixture interaction and documenting or fixing its cause.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devtools, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100