testcontainers / testcontainers/testcontainers-python
Bug: Starting multiple containers at once in parallel causes Conflict by starting ryuk container multiple times due to race condition
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 386
- Avg merge
- 4h 40m
- Merged PRs (30d)
- 1
Description
Describe the bug
Starting two testcontainers at once triggers both of them to call Reaper._create_instance close in time. Both Reaper instances attempt to create the same ryuk container (since SESSION_ID is the same).
This is relevant when using pytest-parallel, which my organization does. I can probably work around the problem by serializing testcontainer start() calls on one thread.
To Reproduce
from testcontainers.postgres import PostgresContainer
import concurrent.futures
def start_postgres_container(port: int):
with PostgresContainer(port=port) as postgres:
print("Postgres is running on port", postgres.get_exposed_port(port))
with concurrent.futures.ThreadPoolExecutor() as executor:
future1 = executor.submit(start_postgres_container, port=5400)
# Uncommenting this reliably avoids the error
# import time
# time.sleep(5)
future2 = executor.submit(start_postgres_container, port=5401)
future1.result()
future2.result()
Runtime environment
Provide a summary of your runtime environment. Which operating system, python version, and docker version are you using? What is the version of testcontainers-python you are using? You can run the following commands to get the relevant information.
# Get the operating system information (on a unix os).
$ uname -a
Darwin xxx 24.1.0 Darwin Kernel Version 24.1.0: Thu Oct 10 21:03:11 PDT 2024; root:xnu-11215.41.3~2/RELEASE_ARM64_T6020 arm64 arm Darwin
$ python --version
Python 3.11.10
$ docker info
# Get all python packages.
$ pip freeze
certifi==2024.8.30
charset-normalizer==3.4.0
docker==7.1.0
idna==3.10
requests==2.32.3
testcontainers==4.8.2
typing_extensions==4.12.2
urllib3==2.2.3
wrapt==1.17.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Reaper._create_instance and run the concurrent.futures reproduction from the issue using two PostgresContainer instances. Done means concurrent starts no longer produce a Conflict from attempting to create the ryuk container multiple times, while both containers start successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100