litestar-org / litestar-org/pytest-databases

fix: serialize cross-service Docker port allocation under xdist

Open
#152 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
59
Forks
11
PR merge metrics
No merged PRs in 30d

Description

## Problem

`DockerService.run()` protects each service name with its own xdist file lock, but different services can still call `client.containers.run(... ports={container_port: None})` concurrently on separate workers.

On rootless Docker, one SQLSpec `pytest -n 4 --dist=loadgroup` run started several native pytest-databases services at once. Docker allocated adjacent host ports 33313-33322, while the MSSQL container failed on 33316:

```text
RootlessKit PortManager.AddPort(): listen tcp4 0.0.0.0:33316:
bind: address already in use
```

There was no second pytest process. The concurrent starts were different service names from the same run. Because `mssql_service` is session-scoped, the one startup failure cascaded into every MSSQL test.

Issue #131 / PR #133 added `host_port` support to `DockerService.run`, but only the PostgreSQL-family fixture surface exposes port overrides. MSSQL, MySQL, Oracle, GizmoSQL, CockroachDB, RustFS, and other native service fixtures still use dynamic allocation and cannot opt into a pinned port.

## Suggested fix

Serialize only the container-creation/port-allocation critical section across service names when xdist is active, using a global file lock under the pytest-databases temp directory. Keep the existing per-name lock for service reuse and lifecycle behavior.

Alternatively, expose an optional host-port fixture consistently for every Docker service, though a global creation lock is less configuration and protects all current and future services.

## Environment

- pytest-databases 0.19.0
- pytest-xdist 3.8.0
- docker-py 7.x
- Linux rootless Docker / RootlessKit
- Python 3.10

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at DockerService.run and trace the existing per-service xdist file lock and the native service fixtures that request dynamic ports. Preserve service reuse and lifecycle behavior while serializing container creation across service names when xdist is active. Validate with a pytest -n 4 --dist=loadgroup run covering MSSQL and other native Docker services.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
databases, devops, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.