pytest-dev / pytest-dev/pytest-xdist
[BUG] Test error when worker amount set to auto
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 287
- Avg merge
- 9h 30m
- Merged PRs (30d)
- 2
Description
First at all, thanks for your time and the develop of this project.
I'm workin with pytest-django and postgres.
Sometimes when i start a test session using -n auto, where it starts with 12 workers, the tests raises an error becouse it fails to drop the previous instaciated db's
In my tests, it only happen when the worker amount is set auto.
Not sure if the issue is caused by pytest-xdist, but i belive so becouse when i run only python -m pytest --no-migrations it works just fine.
These are some of the retuns i'm getting:
python -m pytest --no-migrations -n auto
=================================================================================== test session starts ====================================================================================
platform linux -- Python 3.12.7, pytest-8.4.2, pluggy-1.5.0
django: version: 5.2.7, settings: myproject_settings.test (from option)
rootdir: /home/lucas/projects/myproject
configfile: pyproject.toml
testpaths: tests
plugins: xdist-3.8.0, mock-3.15.1, anyio-4.11.0, Faker-33.0.0, snapshot-0.9.0, django-4.9.0, cov-6.2.1
12 workers [3918 items]
EEEEEEEEEEEEEEEEEEE.EEE..E.......E..EE.....E....E.E..E....E......E..EE....E........E.........................E....E....E...E...E.....E.E...E....E....E..E.E...E.E...EE.....E.....E.. [ 4%]
.EE..E....E...EE...E.E.....EEE...E......E..EE.........E.....E.EE.....E....E....EE........E.....E..EE.....E......E...E................E...E.....EEE.E.....E.EE.E....E..E.E..E....E.E. [ 9%]
..E.E...E...E.E..E..EEE.E....E.EE..E.E....EE.E.E.E.E.E..E..E..E.E....E..E.E...E....E.E................E.....E...E....EE...E....E..EE.E.E...EE..EE.E.E..E.E.EE..E.E..EE...EE..E.E.... [ 13%]
E.E...E.E....EE....E.E..EE......EE..E.E....E..E.EE....E.........E....E.E...EE..EE...E.E.E...E...E.E.E..E....E...EE.E.E.....E.E..E...E..EE.E.E....E.EE.E..EE.E.E.....EE.E.E...E..E... [ 18%]
E..E...E..EE.....E....E...E.EE......E..EE...E..EE.EE.....E..EEE...EEEE.....E.EEE...EEEE..EEE...EE.EEE......E.EE.E.......EEEE....EE.E....E.E........EEE......EE.E.E...E.EEE.....EEEE. [ 22%]
...EEE.E....EEEE.....E.EEE......EEEE.......E..EEE....E..E.E.E.......E..E.EE..........E..E..E..............E.E.....E..E.E.....EE.E..E.EE..EE....EE..EE......E..E..EE.....E..E.EE..... [ 27%]
..E.EE.E....E.EE.E....E..E.E.E.....E..EE..E.....EEE.E...E.EE..E..EE.E....E..E.....E..E...E.E.E..E.E...EE....E.E....E.E.E.E....EE..EE...E.EE..E...E...EE.E.....E..EEE.......E..EE.E.. [ 32%]
...EEEE.....E.EEE.E.EE....E.E......E..E...E..E.EE.E...E.EE...E..E..EEEE.EEE..E.E.EE....EEE...E....EE.E.E...E.EEE.....EEE...E....EEE...E...E.E...E.....E..E..............EE..E....EE. [ 36%]
E.E....EE.E.E....EEE.E.....E.E.EE..E.E.EE....E...E.EE..E..EEE....E...EEE..E.EE.E....E..EEE.......EEE....E.E...E....EE..E...E.EE....E...EEE..E.E.EE...E..E.E.E..E.EE..E..E...EE..E.E. [ 41%]
.E.EEE.EEE..E..EE.E.E..EE.E.....EE.E.E....E....E...EEE.......EE.EE.......E..EE..E......E..EE.E.E.EE..E.EEE.E...EEEE.....E.....EEE....E..EE.E.EEE.E.EEE..E...EEE....EEE.EE.E.E.EE.E.. [ 45%]
.EE.E..E..EEEE.E.EE.E..E.EE..E...EEE..E..EE..EE...E.E.EE....E..EEE.....EEE...E..EE.......EEE..EEE..E.EE..E.EEE..E..E.EE.EEEE..E...E.EE..EEEE.E..EEEEEEE.....EEEE..EE.E....EE.E.E..E. [ 50%]
.E.E..EE.E...EEEEEEE...E..E.EEEE...EE..E.E..EE...E.E.E.E.E.E..EE.E.E...EE.....E..E..E.E...E..E..E..E.......E..E.E..E....E..E......E...EE.......EE..E.....E..EE...E...E.EE....E...E.. [ 55%]
EE..E....E.E.E..E..E..E.E....E...E.EE....E.E.E.E.E...EE..E....E..E..E......E...E......EE.....EE...E.EEE..E.....EE..E..EEE.E.EE.E..E...EEE.E..EEEE....EE.E.EEEE..E......EEE.E...EEEE. [ 59%]
EE.E.E....EE.....E..E..E........E.EE...E....EE..E......E.EE.E....EEE.E.....EEE..E.EE.EE..E..EE..E..E..E..E.E..E.E.EE....E..EE.E..E...EEE...E....E.E.E.E.EE.E..EE.E..EE..EE.E.EEEE.E. [ 64%]
.E.EEE.E.E.EEE..E.EE...E..E.EEEE.EE.EEEE...E.E.EE..E.E..E..EEEE.....EEE.EE...EEE.EEE..EEE..E.EE.E..E..EEEEEE.EEEE.E.E....EE.E..EEE.E..E.E.E.E..E...EE..EEE......EE.E...E...E.EEEEEE. [ 68%]
..EEEEEEE..EEEE...EEE.E..EEEE..E.E..EE...EE.E....E.....EEE.....E....EE......E..EE.E........E.EE............E...EE............E...E.E..E....E...EE.E...E..E...EE.....E...E.E.E.....E. [ 73%]
EE....E...E.E.E....E...E.E.E...........E....E...E.E.........E...E...E.E.........E......E................E......E.......E..E......E............E.E..E.........E.....EE............... [ 78%]
...E......EE..........E.........E.....EE.......E.......E.....E.E....E......E....E.E.....E.....E....EE......E.......EE.E......E....EE.E......E...EEE.....E....E.EE.......E...E....... [ 82%]
..EE......EE....E......E..E.....E.........EE...E...........E.E.....E.......E....E.E....E.......E.....EE.....E....E.....E.E...E......E...E.....E.E.......E..E....EE...........E.E.... [ 87%]
....E.E.............EE....E.....E...........E...E....E.E......................E...E....................E..EE..........E..............E...E............E.........E.....E.......E..... [ 91%]
..E............E.....E.........E...........E..E.....E.......E.......E..E......E.....E......E...E...E.E...E.E..EE.E.EE.EEE.E.EEE..EE..EEEE..E....EE...E..E...E.E..E...E.E....E....... [ 96%]
E..E......E.......E...E..E...E.......EE...EE......EE....EE........E....E.....E.E...E.E..E.E..EE.EEEE.EEEE..E..EEE.E...E.E.EE..EE...E...... [100%]
Plugin: 124523839128928, Hook: pytest_sessionfinish
OSError: cannot send (already closed?)
For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
config.hook.pytest_sessionfinish(
/home/lucas/projects/myproject/.venv/lib/python3.12/site-packages/_pytest/main.py:324: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
ERROR tests/vehicles/test_views.py::TestVehicleSetUpdateView::test_post_404 - django.db.utils.OperationalError: out of shared memory
self = <django.db.backends.utils.CursorWrapper object at 0x790bea824230>
sql = 'CREATE INDEX "index" ON "table" ("id")', params = None
ignored_wrapper_args = (False, {'connection': <DatabaseWrapper vendor='postgresql' alias='default'>, 'cursor': <django.db.backends.utils.CursorWrapper object at 0x790bea824230>})
def _execute(self, sql, params, *ignored_wrapper_args):
# Raise a warning during app initialization (stored_app_configs is only
# ever set during testing).
if not apps.ready and not apps.stored_app_configs:
warnings.warn(self.APPS_NOT_READY_WARNING_MSG, category=RuntimeWarning)
self.db.validate_no_broken_transaction()
with self.db.wrap_database_errors:
if params is None:
# params default might be backend specific.
> return self.cursor.execute(sql)
^^^^^^^^^^^^^^^^^^^^^^^^
E django.db.utils.OperationalError: out of shared memory
E HINT: You might need to increase max_locks_per_transaction.
.venv/lib/python3.12/site-packages/django/db/backends/utils.py:103: OperationalError
and this is the db state after the test finishes
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the failure with python -m pytest --no-migrations -n auto and compare it with the non-xdist command. Start at pytest_sessionfinish and the worker/database teardown behavior, using the reported out of shared memory and cannot send (already closed?) errors as checkpoints; done means automatic workers complete without teardown failures or PostgreSQL lock exhaustion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases, distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100